Hi folks,
Recently been running "Common scenario stress with IO" WHQL test on my
driver (handling a PCIe network card), and got really frustrated that I
was occasionally seeing very random bugchecks where pages of memory got
corrupted. As of about a week ago, we got to the point where it looked
like some hardware was DMAing descriptors into random pages of memory,
but looking at the memory, the descriptors did not match anything that
our hardware would write.
So, scratching heads really hard, we decided to change the test to
enable driver verifier for all drivers. We then had a few more random
crashes with no obvious leads, except corrupted pages after several
hibernate cycles, and *eventually* we got a crash where driver verifier
caught what was going on. Version info and !analyze output included below.
If the MS guys want more detail, I have a full kernel memory dump. It
looks like in order to get it, you need to repeatedly hibernate a 64 bit
vista system with an appropriate raid controller ... we haven't seen it
on any other platforms / hardware combinations.
Version info:
Windows Vista Kernel Version 6000 MP (2 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 6000.16386.amd64fre.vista_rtm.061101-2205
Kernel base = 0xfffff80001800000 PsLoadedModuleList = 0xfffff80001999e90
Debug session time: Thu May 31 16:48:55.482 2007 (GMT+1)
System Uptime: 0 days 5:40:14.283
0: kd> lm v m storport
start end module name
fffff980004aa000 fffff98000500000 storport (pdb symbols)
c:\websymbols\storport.pdb\DBAF4E3FBB5E4C58A5AF45FF1BECE55C1\storport.pdb
Loaded symbol image file: storport.sys
Image path: \SystemRoot\system32\drivers\storport.sys
Image name: storport.sys
Timestamp: Thu Nov 02 09:38:25 2006 (4549BC91)
CheckSum: 00025342
ImageSize: 00056000
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
Crashdump output:
0: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck
Analysis *
*
*
*******************************************************************************
DRIVER_VERIFIER_DMA_VIOLATION (e6)
An illegal DMA operation was attempted by a driver being verified.
Arguments:
Arg1: 000000000000000e, Buffer not locked. DMA transfer has been
attempted with a PAGED buffer.
Arg2: 0000000000000000
Arg3: 0000000000000000
Arg4: 0000000000000000
Debugging Details:
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
BUGCHECK_STR: 0xE6
PROCESS_NAME: System
CURRENT_IRQL: 2
LAST_CONTROL_TRANSFER: from fffff80001c0837d to fffff8000184e250
STACK_TEXT:
fffff9802aff1058 fffff80001c0837d : 00000000000000e6 000000000000000e 0000000000000000 0000000000000000 : nt!KeBugCheckEx
fffff9802aff1060 fffff80001c10a7c : fffff9802aff1140 fffff80001c134da 0000000000000003 fffff80001c0979a :
nt!VerifierBugCheckIfAppropriate+0x3d
fffff9802aff10a0 fffff800018d8792 : fffffa8008b24830 0000000000000004 0000000000000065 0000000000000003 : nt!VfAssert+0xbc
fffff9802aff10e0 fffff80001c226a9 : 0000000000000000 fffffa80054dd000 0000000000000000 0000000c00000000 :
nt!VERIFY_BUFFER_LOCKED+0xa2
fffff9802aff1110 fffff980004ad3d5 : 0000000000004000 fffffa800552f390 fffffa8008b24830 fffffa80054dd000 :
nt!VfBuildScatterGatherList+0x99
fffff9802aff11b0 fffff980004b9b7a : fffffa80055301b0 0000000000000000 fffff9803b5e4f90 fffff9800a440e50 :
storport!RaidAdapterScatterGatherExecute+0x8d
fffff9802aff1210 fffff980004b21cb : fffff9800a440ec8 fffff9800a440ec8 0000000000000000 fffffa80055301b0 :
storport!RaUnitStartIo+0xe2
fffff9802aff1270 fffff980004b949f : fffff9803b5e4f90 0000000000000001 fffffa80055301b0 fffff9800a440e50 :
storport!RaidStartIoPacket+0x1fb
fffff9802aff12f0 fffff980004b9112 : fffffa8008669a70 fffff80001c21452 fffff9800a440f68 fffff80001c23d64 :
storport!RaidUnitSubmitRequest+0x97
fffff9802aff1320 fffff980004b13ae : fffff9800a440e50 0000000000000002 fffff9800a440e50 fffffa8005530060 :
storport!RaUnitScsiIrp+0xc6
fffff9802aff13a0 fffff80001c244e6 : fffff9800a440e50 fffffa8005530060 fffffa80057d8640 fffff9800a440e50 :
storport!RaDriverScsiIrp+0x7e
fffff9802aff13e0 fffff80001c21452 : fffff9800a440f68 0000000000000002 fffffa8000000001 fffffa80057d8640 :
nt!IovCallDriver+0x346
fffff9802aff1420 fffff80001c244e6 : fffff9800a440e50 0000000000000002 fffffa800562daa0 fffff8000185ea7a :
nt!ViFilterDispatchGeneric+0x62
fffff9802aff1450 fffff80001c245c6 : 0000000000000000 fffff9802aff14b0 0000000000000000 fffffa800530e130 :
nt!IovCallDriver+0x346
fffff9802aff1490 fffff98000b07567 : 0000000000000000 0000000000000000 0000000aa7ae8000 0000000aa7ae8000 :
nt!VerifierIoCallDriverStackSafe+0x96
fffff9802aff14d0 fffff98000b06a29 : fffffa800562d1b0 fffffa800562d100 fffff98000101000 fffff80000004000 :
CLASSPNP!ServiceTransferRequest+0x28b
fffff9802aff1560 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa800562d1b0 fffff98050022a60 :
CLASSPNP!ClassReadWrite+0x17d
fffff9802aff15b0 fffff80001c21452 : fffff98050022c08 0000000000000002 fffffa8000000000 fffffa8005c9c010 :
nt!IovCallDriver+0x346
fffff9802aff15f0 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa800562ee20 fffff98050022ce0 :
nt!ViFilterDispatchGeneric+0x62
fffff9802aff1620 fffff80001c245c6 : fffff98050022c50 fffff9802aff1680 fffff80001c21452 fffffa80052abe60 :
nt!IovCallDriver+0x346
fffff9802aff1660 fffff98000ba36ff : fffff98050022c50 fffffa800562eb00 fffff98050022a60 fffff98000000001 :
nt!VerifierIoCallDriverStackSafe+0x96
fffff9802aff16a0 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa800562e9b0 fffff98050022a60 :
partmgr!PmReadWrite+0x133
fffff9802aff16d0 fffff80001c21452 : fffff98050022c98 0000000000000002 fffffa8000000000 fffffa8007224f40 :
nt!IovCallDriver+0x346
fffff9802aff1710 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa800562e450 fffff98050022d70 :
nt!ViFilterDispatchGeneric+0x62
fffff9802aff1740 fffff80001c245c6 : fffff98050022ce0 fffff9802aff17a0 fffff80001c21452 fffffa8006b72990 :
nt!IovCallDriver+0x346
fffff9802aff1780 fffff9800020a8ca : fffff98050022ce0 fffff98050022a60 fffffa800563b1b0 fffff98000000001 :
nt!VerifierIoCallDriverStackSafe+0x96
fffff9802aff17c0 fffff80001c244e6 : fffff98050022a60 fffffa800563b060 fffffa8004b55390 fffff98050022a60 :
volmgr!VmReadWrite+0x1d6
fffff9802aff17f0 fffff80001c21452 : fffff98050022d28 0000000000000002 fffffa8000000000 fffffa8004b55390 :
nt!IovCallDriver+0x346
fffff9802aff1830 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa800567de20 fffff80001c23d64 :
nt!ViFilterDispatchGeneric+0x62
fffff9802aff1860 fffff80001c245c6 : fffffa8005688ba0 fffff9802aff18c0 fffffa800568f5b0 fffffa80073e5f40 :
nt!IovCallDriver+0x346
fffff9802aff18a0 fffff98000b48a40 : fffffa8005688ba0 fffffa8005688a50 fffff98050022a60 fffff98000000000 :
nt!VerifierIoCallDriverStackSafe+0x96
fffff9802aff18e0 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa8005688a50 fffff98050022a60 :
fvevol!FveFilterRundownWrite+0x158
fffff9802aff1920 fffff80001c21452 : fffff98050022db8 0000000000000002 fffffa8000000000 fffffa8004d10270 :
nt!IovCallDriver+0x346
fffff9802aff1960 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa800568f040 fffffa800568f700 :
nt!ViFilterDispatchGeneric+0x62
fffff9802aff1990 fffff80001c245c6 : 0000000000000000 fffff9802aff19f0 fffffa8000000000 fffffa8004bbd6e0 :
nt!IovCallDriver+0x346
fffff9802aff19d0 fffff98000b68fc0 : 0000000000000000 fffff98050022e48 fffff98050022a60 fffff98050022a60 :
nt!VerifierIoCallDriverStackSafe+0x96
fffff9802aff1a10 fffff80001c244e6 : fffff98050022a00 fffffa800568f5b0 0000000000000000 0000000000000000 :
ecache!EcDispatchReadWrite+0x884
fffff9802aff1b40 fffff80001c21452 : fffff98050022e48 0000000000000002 fffffa8000000000 fffffa8006b93d50 :
nt!IovCallDriver+0x346
fffff9802aff1b80 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa800568f390 fffff98050022ed8 :
nt!ViFilterDispatchGeneric+0x62
fffff9802aff1bb0 fffff80001c245c6 : fffff98050022e90 fffff9802aff1c10 fffff98050022a60 fffffa800589be60 :
nt!IovCallDriver+0x346
fffff9802aff1bf0 fffff98000bc511e : fffff98050022e90 fffff98050022a60 fffffa800569c190 fffffa800569c190 :
nt!VerifierIoCallDriverStackSafe+0x96
fffff9802aff1c30 fffff80001c244e6 : fffff98050022a00 fffff98050022a60 0000000000000002 fffffa800569c040 :
volsnap!VolSnapWrite+0x46a
fffff9802aff1c80 fffff80001c21452 : fffff98050022ed8 0000000000000002 fffffa8000000000 fffffa8004e03e20 :
nt!IovCallDriver+0x346
fffff9802aff1cc0 fffff80001c244e6 : fffff98050022a60 0000000000000002 fffffa800569ce20 b5b5b5b5b5b5b5b5 :
nt!ViFilterDispatchGeneric+0x62
fffff9802aff1cf0 fffff9800089feb6 : fffff980014d9ff8 fffff980014da0d0 b5b5b5b5b5b5b5b5 fffffa8006b6b9a0 :
nt!IovCallDriver+0x346
fffff9802aff1d30 fffff800018388f7 : b5b5b5b5b5b5b5b5 b5b5b5b5b5b5b5b5 b5b5b5b5b5b5b5b5 b5b5b5b5b5b5b5b5 :
Ntfs!NtfsStorageDriverCallout+0x16
fffff9802aff1d60 fffff800018388b5 : 0000000000000000 0000000000000000 0000000000000000 fffff800018380ce :
nt!KxSwitchKernelStackCallout+0x27
fffff980014d9f10 fffff800018380ce : 0000000000000000 fffffa80065fe000 000000004d7d2e00 0000000000004000 :
nt!KiSwitchKernelStackContinue
fffff980014d9f30 fffff9800089d4b9 : fffff9800089fea0 fffff980014d9ff8 0000000000000000 fffff980014da130 :
nt!KeExpandKernelStackAndCalloutEx+0x12e
fffff980014d9fc0 fffff9800089cbca : fffff9804d7d2e50 fffff980014da080 fffff9801fdbec80 0000000000000008 :
Ntfs!NtfsMultipleAsync+0xf9
fffff980014da030 fffff98000894896 : fffff9804d7d2e50 fffff98050022a60 fffff9804d7d2e00 fffff800018eb300 :
Ntfs!NtfsNonCachedIo+0x23a
fffff980014da200 fffff9800089b597 : fffff9804d7d2e50 fffff98050022a60 fffff980014da400 fffff980014da400 :
Ntfs!NtfsCommonWrite+0xd0e
fffff980014da420 fffff80001c244e6 : fffff98050022a60 fffff98050022a60 fffffa800577b030 fffffa8007ca06b0 :
Ntfs!NtfsFsdWrite+0x1c7
fffff980014da4e0 fffff9800046821a : fffff98050022fb0 fffff98050022a60 fffffa8008ddb900 fffffa8007ca06b0 :
nt!IovCallDriver+0x346
fffff980014da520 fffff98000468691 : fffffa80057483e0 fffffa80057483e0 fffffa8005748300 0000000000000002 :
fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x20a
fffff980014da590 fffff80001c244e6 : fffff98050022a60 fffffa80057483e0 fffffa8003cc0bd0 0000000000000000 :
fltmgr!FltpDispatch+0xd1
fffff980014da5f0 fffff80001abdd6f : fffffa80057483e0 0000000000000001 0000000000000011 fffffa8004c05b90 :
nt!IovCallDriver+0x346
fffff980014da630 fffff80001ac88c0 : 0000000000000000 fffff980014da890 fffffa80085312f0 fffff980014da720 :
nt!IopSynchronousServiceTail+0x12f
fffff980014da6a0 fffff8000184dcf3 : fffff6fc0000d811 0000000000000000 0000000000000000 0000000000000000 : nt!NtWriteFile+0x49f
fffff980014da7a0 fffff8000184e200 : fffff80001a7380b 0000000000000000 fffff80001ce7da0 000000004bf0f474 :
nt!KiSystemServiceCopyEnd+0x13
fffff980014da9a8 fffff80001a7380b : 0000000000000000 fffff80001ce7da0 000000004bf0f474 0000000000000000 : nt!KiServiceLinkage
fffff980014da9b0 fffff80001adf5c5 : fffffa800532ac50 fffffa80054dd000 00000000c0000001 fffff80001b0a0d0 :
nt!EtwpFlushBufferToLogfile+0x8b
fffff980014daa20 fffff80001a92ef9 : fffffa80054dd000 0000000000000000 0000000000000000 fffff980014dab50 :
nt!EtwpFlushBuffer+0xb5
fffff980014daa60 fffff80001a908bf : fffffa8005320000 0000000000000000 0000000000000000 000000000000000b :
nt!EtwpFlushActiveBuffers+0x2c9
fffff980014dacf0 fffff80001ae199b : 0000000000000000 fffffa800532a7f0 0000000000000080 fffffa800532ac50 : nt!EtwpLogger+0x21f
fffff980014dad50 fffff80001834b86 : fffff80001949880 fffffa800532a7f0 fffffa8004b29040 fffffa8004aa8478 :
nt!PspSystemThreadStartup+0x5b
fffff980014dad80 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 :
nt!KiStartSystemThread+0x16
STACK_COMMAND: kb
FOLLOWUP_IP:
storport!RaidAdapterScatterGatherExecute+8d
fffff980`004ad3d5 3d230000c0 cmp eax,0C0000023h
SYMBOL_STACK_INDEX: 5
SYMBOL_NAME: storport!RaidAdapterScatterGatherExecute+8d
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: storport
IMAGE_NAME: storport.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 4549bc91
FAILURE_BUCKET_ID: X64_0xE6_VRF_storport!RaidAdapterScatterGatherExecute+8d
BUCKET_ID: X64_0xE6_VRF_storport!RaidAdapterScatterGatherExecute+8d
Followup: MachineOwner