I saw a crash when running under Verifier with delay fuzzing, and suspect it’s a legit bug somewhere.
It appears, some NDIS_SPINLOCK might have gotten released twice in a row: first by a legit NdisReleaseSpinLock, then got acquired by a waiter on a different processor (with PASSIVE_LEVEL original IRQL), then got released extra time by the first processor. As a result, it might have went to PASSIVE_LEVEL instead of staying on DISPATCH_LEVEL, and it screwed KiExecuteDpc thread. Then VerifierExt!CuzzSchedule saw that it’s on PASSIVE_LEVEL and saw fit to issue Sleep(), which caused the bugcheck.
Is there some verifier setting that would check that a spinlock gets released by a wrong processor?
KDTARGET: Refreshing KD connection
*** Fatal System Error: 0x000000b8
(0xFFFFB3896D7F1040,0xFFFFDB80B0E0CFC0,0x0000000000000000,0x0000000000000000)
Break instruction exception - code 80000003 (first chance)
A fatal system error has occurred.
Debugger entered on first try; Bugcheck callbacks have not been invoked.
A fatal system error has occurred.
nt!DbgBreakPointWithStatus:
fffff802`94557a60 cc int 3
2: kd> !analyze -v
Connected to Windows 10 14393 x64 target at (Sun Aug 28 11:35:41.150 2016 (UTC - 7:00)), ptr64 TRUE
Loading Kernel Symbols
…
…
Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.
…
Loading User Symbols
Loading unloaded module list
…Unable to enumerate user-mode unloaded modules, Win32 error 0n30
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
ATTEMPTED_SWITCH_FROM_DPC (b8)
A wait operation, attach process, or yield was attempted from a DPC routine.
This is an illegal operation and the stack track will lead to the offending
code and original DPC routine.
Arguments:
Arg1: ffffb3896d7f1040, Original thread which is the cause of the failure
Arg2: ffffdb80b0e0cfc0, New thread
Arg3: 0000000000000000, Stack address of the original thread
Arg4: 0000000000000000
Debugging Details:
DUMP_CLASS: 1
DUMP_QUALIFIER: 0
BUILD_VERSION_STRING: 14393.0.amd64fre.rs1_release.160715-1616
DUMP_TYPE: 0
BUGCHECK_P1: ffffb3896d7f1040
BUGCHECK_P2: ffffdb80b0e0cfc0
BUGCHECK_P3: 0
BUGCHECK_P4: 0
FAULTING_THREAD: 6d7f1040
CPU_COUNT: 14
CPU_MHZ: 8fc
CPU_VENDOR: GenuineIntel
CPU_FAMILY: 6
CPU_MODEL: 3f
CPU_STEPPING: 2
CPU_MICROCODE: 6,3f,2,0 (F,M,S,R) SIG: 37’00000000 (cache) 37’00000000 (init)
DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT
BUGCHECK_STR: 0xB8
PROCESS_NAME: System
CURRENT_IRQL: 2
ANALYSIS_SESSION_HOST: CCX-DEVS-SRVR
ANALYSIS_SESSION_TIME: 08-28-2016 11:35:48.0144
ANALYSIS_VERSION: 10.0.14321.1024 amd64fre
LAST_CONTROL_TRANSFER: from fffff80294441cdc to fffff80294556da6
STACK_TEXT:
ffffdb80b75eaf40 fffff80294441cdc : 0000000000000000 0000000000000001 00000000000000d5 fffff8029447e720 : nt!KiSwapContext+0x76
ffffdb80b75eb080 fffff8029444177f : ffffb3896d7f1040 ffffb3896d7f1140 0000000000000000 0000000000000000 : nt!KiSwapThread+0x17c
ffffdb80b75eb130 fffff80294496eb6 : ffffb38900000000 ffffb38900000000 ffffdb8000000034 ffffdb80b75eb288 : nt!KiCommitThreadWait+0x14f
ffffdb80b75eb1d0 fffff8073ea16847 : 00000000000001f4 00000000002245cd 0000000000000000 0000000000000000 : nt!KeDelayExecutionThread+0x106
ffffdb80b75eb250 fffff8073ea1720a : 0000000000000001 fffffffffffffe0c 0000000000000004 fffff80294433d53 : VerifierExt!NativeSleep+0x43
ffffdb80b75eb280 fffff8073ea16446 : fffff8073ea38580 00000000000f4240 0000000000000032 ffffb38900010000 : VerifierExt!PCTStep+0x3f2
ffffdb80b75eb300 fffff8073ea16551 : fffff8073ea38580 00000000000000db 000000004d737356 ffffdb8000000000 : VerifierExt!CuzzScheduleInternal2+0xea
ffffdb80b75eb340 fffff8073ea06a4d : 0000000000000000 ffffdb80b75eb480 0000000000000000 ffffb38970c72008 : VerifierExt!CuzzSchedule+0xb1
ffffdb80b75eb380 fffff80294b1ae8a : ffffb38970c72008 0000000000000000 0000000000000000 ffffb3896d62c1b0 : VerifierExt!KeAcquireSpinLockRaiseToDpc_wrapper+0x4d
ffffdb80b75eb3c0 fffff80294b19eb6 : ffff85076062eda0 0000000000000280 0000000000000278 ffffb38900000000 : nt!ViKeAcquireSpinLockRaiseToDpcCommon+0x36
ffffdb80b75eb3f0 fffff8073edd56cb : ffff85076062eda0 ffffdb80b75eb480 0000000000000000 0000000000000011 : nt!VerifierKeAcquireSpinLockRaiseToDpc+0x12
ffffdb80b75eb430 fffff8073f626dd4 : 0000000000000000 0000000000000000 0000000000000000 0000000000000030 : NDIS!NdisAllocateCloneNetBufferList+0x41b
ffffdb80b75eb5d0 fffff8073f6269a2 : 0000000000000000 0000000000000000 0000000000000000 0000000000000001 : vmswitch!VmsNblHelperCreateCloneNbl+0xc4
ffffdb80b75eb7c0 fffff8073f62d504 : ffff850700000000 ffffb3896d620000 ffffdb80b75eb968 0000000000008000 : vmswitch!VmsMpNicPvtPacketForward+0x362
ffffdb80b75eb860 fffff8073f62c9d3 : 0000000000000000 ffffdb80b75ebb00 ffff8507607def60 00000000000002c0 : vmswitch!VmsRouterDeliverNetBufferLists+0xa14
ffffdb80b75eba20 fffff8073edd392e : 0000000000000000 ffffdb80b75ebb10 ffffb3897051a1a0 0000000000000000 : vmswitch!VmsExtPtReceiveNetBufferLists+0x1a3
ffffdb80b75eba90 fffff8073edf979c : 0000000000008000 0000000000000000 ffff850700000000 fffff80200000001 : NDIS!ndisMIndicateNetBufferListsToOpen+0x11e
ffffdb80b75ebb50 fffff8073ede05a3 : ffffb3897051a1a0 0000000000000000 ffff850748e3ede0 0000000000000009 : NDIS!ndisMTopReceiveNetBufferLists+0x265fc
ffffdb80b75ebc60 fffff8073ee002b2 : 0000000000000000 ffff85074eb0ac70 ffff850748e3ede0 ffffdb80b75ebce8 : NDIS!ndisInvokeNextReceiveHandler+0x4f
ffffdb80b75ebd30 fffff8073edde674 : ffff85074eb0ac70 0000000000000001 0000000000000000 0000000000000001 : NDIS!ndisFilterIndicateReceiveNetBufferLists+0x21c12
ffffdb80b75ebdd0 fffff8073f629c42 : 0000000000000001 ffffdb80b75ebe81 ffffb38970510000 0000000000008000 : NDIS!NdisFIndicateReceiveNetBufferLists+0x54
ffffdb80b75ebe10 fffff8073edd3e97 : ffffb389704ec000 000000000000ff00 ffffdb8000000001 0000000000000000 : vmswitch!VmsExtFilterEgressFilterNetBufferLists+0x352
ffffdb80b75ebee0 fffff8073edee68e : ffffdb80b75ebfd0 0000000000000008 ffffdb80b6ccc260 fffff8073f956e09 : NDIS!ndisCallReceiveHandler+0x47
ffffdb80b75ebf30 fffff80294554e27 : ffffdb80b6ccc3f8 0000000000000000 0000000000000012 0000000000000004 : NDIS!ndisDataPathExpandStackCallback+0x3e
ffffdb80b75ebf80 fffff80294554ded : 0000000000004c02 ffffdb80b75ec000 ffffb3896d7f1040 fffff8029446d1e4 : nt!KxSwitchKernelStackCallout+0x27
ffffdb80b1154720 fffff8029446d1e4 : fffff80700000006 0000000000004c00 fffff8073f7454cc fffff80294433dc4 : nt!KiSwitchKernelStackContinue
ffffdb80b1154740 fffff8029446cf56 : ffffdb80b1150000 0000000000004ccc 0000000000000000 ffffdb80b11547c0 : nt!KiExpandKernelStackAndCalloutOnStackSegment+0x134
ffffdb80b11547c0 fffff8029446ce1f : ffffdb80b11549b0 ffffdb80b11549f8 0000000000008000 0000000000000000 : nt!KiExpandKernelStackAndCalloutSwitchStack+0xa6
ffffdb80b1154820 fffff8073ede2a75 : ffff85074eb0ac70 fffff8073f6298e0 000000000001cdb8 ffffb3897051a1a0 : nt!KeExpandKernelStackAndCalloutInternal+0x2f
ffffdb80b1154870 fffff8073edf9518 : 0000000000008000 ffffdb80b11549b0 000000000000003f fffff80294433dc4 : NDIS!ndisExpandStack+0x19
ffffdb80b11548b0 fffff8073f648fba : 000002020018002b 000095dc00001f80 00007ffffffeffff ffffb3896d7f1040 : NDIS!NdisMIndicateReceiveNetBufferLists+0x26f68
ffffdb80b1154aa0 fffff8073f648c6d : ffffb38971562000 ffffb38970510000 0000000000000000 0000fc00005e0001 : vmswitch!VmsExtMpIndicatePackets+0x1ca
ffffdb80b11550d0 fffff8073edddb3e : 0000004200000800 0000000000000042 ffffb3897051a102 0000000100000000 : vmswitch!VmsExtMpSendNetBufferLists+0x46d
ffffdb80b1155270 fffff8073eddda3e : 0000000000000000 ffff850748e3ede0 ffff850748e3ede0 fffff80294433d53 : NDIS!ndisMSendNBLToMiniportInternal+0xee
ffffdb80b1155320 fffff8073edee6da : ffffdb80b1156000 ffffdb80b1155374 ffffb389704ec000 0000000000000005 : NDIS!ndisMSendNBLToMiniport+0xe
ffffdb80b1155360 fffff8073edde22d : 0000fc00005e0001 ffff85074eb0ac70 0000000000000024 ffffdb80b1155450 : NDIS!ndisInvokeNextSendHandler+0x46
ffffdb80b1155440 fffff8073f6293d6 : ffff850748e3ede0 ffffdb80b100ff00 ffff850748e3ed00 0000fc00005e0001 : NDIS!NdisFSendNetBufferLists+0x2ed
ffffdb80b11554c0 fffff8073edd4bd4 : ffffb389704ec000 0000000000000001 0000000000000000 0000000000000024 : vmswitch!VmsExtFilterIngressFilterNetBufferLists+0x4d6
ffffdb80b1155600 fffff8073edee68e : ffff90f1e9430e6c ffffb389705c71a0 0000000000000001 fffff802944a248e : NDIS!ndisCallSendHandler+0x44
ffffdb80b1155650 fffff8029446ce75 : ffffdb80b11557f0 ffffdb80b11557a8 fffff8073f628ef0 fffff80294433e0b : NDIS!ndisDataPathExpandStackCallback+0x3e
ffffdb80b11556a0 fffff8073edd4b39 : ffff850748e3ede0 ffff85074eb0ac70 ffffb3897051a1a0 0000000000000000 : nt!KeExpandKernelStackAndCalloutInternal+0x85
ffffdb80b11556f0 fffff8073f631e0e : ffff85074eb32c10 ffff850748e3ede0 ffffb38900000000 ffff850748e3ede0 : NDIS!NdisSendNetBufferLists+0x559
ffffdb80b1155820 fffff8073f635502 : ffffb389704b8701 0000000000000001 ffffb389704b8720 ffffb389704b8700 : vmswitch!VmsExtPtRouteNetBufferLists+0x40e
ffffdb80b11558f0 fffff80294440001 : ffffdb80b0e02f28 0000000000000000 ffffdb80b1155a60 0000000000000000 : vmswitch!VmsPtHostRssStandardThreadedDpc+0x162
ffffdb80b1155960 fffff80294534823 : ffffdb80b0e00180 0000000000000080 ffffdb80b0e02f6e ffffdb80b0e00180 : nt!KiExecuteAllDpcs+0x2b1
ffffdb80b1155ab0 fffff802944a44bd : 0000000000000000 ffffb3896d7f1040 0000000000000080 0000000000000000 : nt!KiExecuteDpc+0x93
ffffdb80b1155c10 fffff80294557456 : ffffdb80b0e00180 ffffb3896d7f1040 fffff802944a447c 0000000000000000 : nt!PspSystemThreadStartup+0x41
ffffdb80b1155c60 0000000000000000 : ffffdb80b1156000 ffffdb80b1150000 0000000000000000 0000000000000000 : nt!KiStartSystemThread+0x16
STACK_COMMAND: .thread 0xffffb3896d7f1040 ; kb
THREAD_SHA1_HASH_MOD_FUNC: 764fccbe59568ce2929fd4238ae4ce1a75ef30c8
THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 05b964782f1fa0304a5f80b5c7ff8eda7085d21b
THREAD_SHA1_HASH_MOD: eba13483c685b5460f071b095c39d72751763638
FOLLOWUP_IP:
vmswitch!VmsNblHelperCreateCloneNbl+c4
fffff807`3f626dd4 488bf8 mov rdi,rax
FAULT_INSTR_CODE: 48f88b48
SYMBOL_STACK_INDEX: c
SYMBOL_NAME: vmswitch!VmsNblHelperCreateCloneNbl+c4
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: vmswitch
IMAGE_NAME: vmswitch.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 578999dc
BUCKET_ID_FUNC_OFFSET: c4
FAILURE_BUCKET_ID: 0xB8_VRF_vmswitch!VmsNblHelperCreateCloneNbl
BUCKET_ID: 0xB8_VRF_vmswitch!VmsNblHelperCreateCloneNbl
PRIMARY_PROBLEM_CLASS: 0xB8_VRF_vmswitch!VmsNblHelperCreateCloneNbl
TARGET_TIME: 2016-08-28T17:31:05.000Z
OSBUILD: 14393
OSSERVICEPACK: 0
SERVICEPACK_NUMBER: 0
OS_REVISION: 0
SUITE_MASK: 402
PRODUCT_TYPE: 3
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
OSEDITION: Windows 10 Server Enterprise TerminalServer DataCenter SingleUserTS
OS_LOCALE:
USER_LCID: 0
OSBUILD_TIMESTAMP: 2016-07-15 19:16:17
BUILDDATESTAMP_STR: 160715-1616
BUILDLAB_STR: rs1_release
BUILDOSVER_STR: 10.0.14393.0.amd64fre.rs1_release.160715-1616
ANALYSIS_SESSION_ELAPSED_TIME: 4939
ANALYSIS_SOURCE: KM
FAILURE_ID_HASH_STRING: km:0xb8_vrf_vmswitch!vmsnblhelpercreateclonenbl
FAILURE_ID_HASH: {79380350-b249-c7d2-4a4b-b945082d85a0}
Followup: MachineOwner
2: kd> .thread 0xffffb3896d7f1040
Implicit thread is now ffffb389`6d7f1040