Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

NDIS and DriverVerifier

Alex_GrigAlex_Grig Member Posts: 3,238
I saw a crash when running under Verifier with delay fuzzing, and suspect it's a legit bug somewhere.

It appears, some NDIS_SPINLOCK might have gotten released twice in a row: first by a legit NdisReleaseSpinLock, then got acquired by a waiter on a different processor (with PASSIVE_LEVEL original IRQL), then got released extra time by the first processor. As a result, it might have went to PASSIVE_LEVEL instead of staying on DISPATCH_LEVEL, and it screwed KiExecuteDpc thread. Then VerifierExt!CuzzSchedule saw that it's on PASSIVE_LEVEL and saw fit to issue Sleep(), which caused the bugcheck.

Is there some verifier setting that would check that a spinlock gets released by a wrong processor?


KDTARGET: Refreshing KD connection

*** Fatal System Error: 0x000000b8
(0xFFFFB3896D7F1040,0xFFFFDB80B0E0CFC0,0x0000000000000000,0x0000000000000000)

Break instruction exception - code 80000003 (first chance)

A fatal system error has occurred.
Debugger entered on first try; Bugcheck callbacks have not been invoked.

A fatal system error has occurred.

nt!DbgBreakPointWithStatus:
fffff802`94557a60 cc int 3
2: kd> !analyze -v
Connected to Windows 10 14393 x64 target at (Sun Aug 28 11:35:41.150 2016 (UTC - 7:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
................................................................


Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

............................
Loading User Symbols

Loading unloaded module list
.......Unable to enumerate user-mode unloaded modules, Win32 error 0n30
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

ATTEMPTED_SWITCH_FROM_DPC (b8)
A wait operation, attach process, or yield was attempted from a DPC routine.
This is an illegal operation and the stack track will lead to the offending
code and original DPC routine.
Arguments:
Arg1: ffffb3896d7f1040, Original thread which is the cause of the failure
Arg2: ffffdb80b0e0cfc0, New thread
Arg3: 0000000000000000, Stack address of the original thread
Arg4: 0000000000000000

Debugging Details:
------------------


DUMP_CLASS: 1

DUMP_QUALIFIER: 0

BUILD_VERSION_STRING: 14393.0.amd64fre.rs1_release.160715-1616

DUMP_TYPE: 0

BUGCHECK_P1: ffffb3896d7f1040

BUGCHECK_P2: ffffdb80b0e0cfc0

BUGCHECK_P3: 0

BUGCHECK_P4: 0

FAULTING_THREAD: 6d7f1040

CPU_COUNT: 14

CPU_MHZ: 8fc

CPU_VENDOR: GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3f

CPU_STEPPING: 2

CPU_MICROCODE: 6,3f,2,0 (F,M,S,R) SIG: 37'00000000 (cache) 37'00000000 (init)

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: 0xB8

PROCESS_NAME: System

CURRENT_IRQL: 2

ANALYSIS_SESSION_HOST: CCX-DEVS-SRVR

ANALYSIS_SESSION_TIME: 08-28-2016 11:35:48.0144

ANALYSIS_VERSION: 10.0.14321.1024 amd64fre

LAST_CONTROL_TRANSFER: from fffff80294441cdc to fffff80294556da6

STACK_TEXT:
ffffdb80`b75eaf40 fffff802`94441cdc : 00000000`00000000 00000000`00000001 00000000`000000d5 fffff802`9447e720 : nt!KiSwapContext+0x76
ffffdb80`b75eb080 fffff802`9444177f : ffffb389`6d7f1040 ffffb389`6d7f1140 00000000`00000000 00000000`00000000 : nt!KiSwapThread+0x17c
ffffdb80`b75eb130 fffff802`94496eb6 : ffffb389`00000000 ffffb389`00000000 ffffdb80`00000034 ffffdb80`b75eb288 : nt!KiCommitThreadWait+0x14f
ffffdb80`b75eb1d0 fffff807`3ea16847 : 00000000`000001f4 00000000`002245cd 00000000`00000000 00000000`00000000 : nt!KeDelayExecutionThread+0x106
ffffdb80`b75eb250 fffff807`3ea1720a : 00000000`00000001 ffffffff`fffffe0c 00000000`00000004 fffff802`94433d53 : VerifierExt!NativeSleep+0x43
ffffdb80`b75eb280 fffff807`3ea16446 : fffff807`3ea38580 00000000`000f4240 00000000`00000032 ffffb389`00010000 : VerifierExt!PCTStep+0x3f2
ffffdb80`b75eb300 fffff807`3ea16551 : fffff807`3ea38580 00000000`000000db 00000000`4d737356 ffffdb80`00000000 : VerifierExt!CuzzScheduleInternal2+0xea
ffffdb80`b75eb340 fffff807`3ea06a4d : 00000000`00000000 ffffdb80`b75eb480 00000000`00000000 ffffb389`70c72008 : VerifierExt!CuzzSchedule+0xb1
ffffdb80`b75eb380 fffff802`94b1ae8a : ffffb389`70c72008 00000000`00000000 00000000`00000000 ffffb389`6d62c1b0 : VerifierExt!KeAcquireSpinLockRaiseToDpc_wrapper+0x4d
ffffdb80`b75eb3c0 fffff802`94b19eb6 : ffff8507`6062eda0 00000000`00000280 00000000`00000278 ffffb389`00000000 : nt!ViKeAcquireSpinLockRaiseToDpcCommon+0x36
ffffdb80`b75eb3f0 fffff807`3edd56cb : ffff8507`6062eda0 ffffdb80`b75eb480 00000000`00000000 00000000`00000011 : nt!VerifierKeAcquireSpinLockRaiseToDpc+0x12
ffffdb80`b75eb430 fffff807`3f626dd4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000030 : NDIS!NdisAllocateCloneNetBufferList+0x41b
ffffdb80`b75eb5d0 fffff807`3f6269a2 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000001 : vmswitch!VmsNblHelperCreateCloneNbl+0xc4
ffffdb80`b75eb7c0 fffff807`3f62d504 : ffff8507`00000000 ffffb389`6d620000 ffffdb80`b75eb968 00000000`00008000 : vmswitch!VmsMpNicPvtPacketForward+0x362
ffffdb80`b75eb860 fffff807`3f62c9d3 : 00000000`00000000 ffffdb80`b75ebb00 ffff8507`607def60 00000000`000002c0 : vmswitch!VmsRouterDeliverNetBufferLists+0xa14
ffffdb80`b75eba20 fffff807`3edd392e : 00000000`00000000 ffffdb80`b75ebb10 ffffb389`7051a1a0 00000000`00000000 : vmswitch!VmsExtPtReceiveNetBufferLists+0x1a3
ffffdb80`b75eba90 fffff807`3edf979c : 00000000`00008000 00000000`00000000 ffff8507`00000000 fffff802`00000001 : NDIS!ndisMIndicateNetBufferListsToOpen+0x11e
ffffdb80`b75ebb50 fffff807`3ede05a3 : ffffb389`7051a1a0 00000000`00000000 ffff8507`48e3ede0 00000000`00000009 : NDIS!ndisMTopReceiveNetBufferLists+0x265fc
ffffdb80`b75ebc60 fffff807`3ee002b2 : 00000000`00000000 ffff8507`4eb0ac70 ffff8507`48e3ede0 ffffdb80`b75ebce8 : NDIS!ndisInvokeNextReceiveHandler+0x4f
ffffdb80`b75ebd30 fffff807`3edde674 : ffff8507`4eb0ac70 00000000`00000001 00000000`00000000 00000000`00000001 : NDIS!ndisFilterIndicateReceiveNetBufferLists+0x21c12
ffffdb80`b75ebdd0 fffff807`3f629c42 : 00000000`00000001 ffffdb80`b75ebe81 ffffb389`70510000 00000000`00008000 : NDIS!NdisFIndicateReceiveNetBufferLists+0x54
ffffdb80`b75ebe10 fffff807`3edd3e97 : ffffb389`704ec000 00000000`0000ff00 ffffdb80`00000001 00000000`00000000 : vmswitch!VmsExtFilterEgressFilterNetBufferLists+0x352
ffffdb80`b75ebee0 fffff807`3edee68e : ffffdb80`b75ebfd0 00000000`00000008 ffffdb80`b6ccc260 fffff807`3f956e09 : NDIS!ndisCallReceiveHandler+0x47
ffffdb80`b75ebf30 fffff802`94554e27 : ffffdb80`b6ccc3f8 00000000`00000000 00000000`00000012 00000000`00000004 : NDIS!ndisDataPathExpandStackCallback+0x3e
ffffdb80`b75ebf80 fffff802`94554ded : 00000000`00004c02 ffffdb80`b75ec000 ffffb389`6d7f1040 fffff802`9446d1e4 : nt!KxSwitchKernelStackCallout+0x27
ffffdb80`b1154720 fffff802`9446d1e4 : fffff807`00000006 00000000`00004c00 fffff807`3f7454cc fffff802`94433dc4 : nt!KiSwitchKernelStackContinue
ffffdb80`b1154740 fffff802`9446cf56 : ffffdb80`b1150000 00000000`00004ccc 00000000`00000000 ffffdb80`b11547c0 : nt!KiExpandKernelStackAndCalloutOnStackSegment+0x134
ffffdb80`b11547c0 fffff802`9446ce1f : ffffdb80`b11549b0 ffffdb80`b11549f8 00000000`00008000 00000000`00000000 : nt!KiExpandKernelStackAndCalloutSwitchStack+0xa6
ffffdb80`b1154820 fffff807`3ede2a75 : ffff8507`4eb0ac70 fffff807`3f6298e0 00000000`0001cdb8 ffffb389`7051a1a0 : nt!KeExpandKernelStackAndCalloutInternal+0x2f
ffffdb80`b1154870 fffff807`3edf9518 : 00000000`00008000 ffffdb80`b11549b0 00000000`0000003f fffff802`94433dc4 : NDIS!ndisExpandStack+0x19
ffffdb80`b11548b0 fffff807`3f648fba : 00000202`0018002b 000095dc`00001f80 00007fff`fffeffff ffffb389`6d7f1040 : NDIS!NdisMIndicateReceiveNetBufferLists+0x26f68
ffffdb80`b1154aa0 fffff807`3f648c6d : ffffb389`71562000 ffffb389`70510000 00000000`00000000 0000fc00`005e0001 : vmswitch!VmsExtMpIndicatePackets+0x1ca
ffffdb80`b11550d0 fffff807`3edddb3e : 00000042`00000800 00000000`00000042 ffffb389`7051a102 00000001`00000000 : vmswitch!VmsExtMpSendNetBufferLists+0x46d
ffffdb80`b1155270 fffff807`3eddda3e : 00000000`00000000 ffff8507`48e3ede0 ffff8507`48e3ede0 fffff802`94433d53 : NDIS!ndisMSendNBLToMiniportInternal+0xee
ffffdb80`b1155320 fffff807`3edee6da : ffffdb80`b1156000 ffffdb80`b1155374 ffffb389`704ec000 00000000`00000005 : NDIS!ndisMSendNBLToMiniport+0xe
ffffdb80`b1155360 fffff807`3edde22d : 0000fc00`005e0001 ffff8507`4eb0ac70 00000000`00000024 ffffdb80`b1155450 : NDIS!ndisInvokeNextSendHandler+0x46
ffffdb80`b1155440 fffff807`3f6293d6 : ffff8507`48e3ede0 ffffdb80`b100ff00 ffff8507`48e3ed00 0000fc00`005e0001 : NDIS!NdisFSendNetBufferLists+0x2ed
ffffdb80`b11554c0 fffff807`3edd4bd4 : ffffb389`704ec000 00000000`00000001 00000000`00000000 00000000`00000024 : vmswitch!VmsExtFilterIngressFilterNetBufferLists+0x4d6
ffffdb80`b1155600 fffff807`3edee68e : ffff90f1`e9430e6c ffffb389`705c71a0 00000000`00000001 fffff802`944a248e : NDIS!ndisCallSendHandler+0x44
ffffdb80`b1155650 fffff802`9446ce75 : ffffdb80`b11557f0 ffffdb80`b11557a8 fffff807`3f628ef0 fffff802`94433e0b : NDIS!ndisDataPathExpandStackCallback+0x3e
ffffdb80`b11556a0 fffff807`3edd4b39 : ffff8507`48e3ede0 ffff8507`4eb0ac70 ffffb389`7051a1a0 00000000`00000000 : nt!KeExpandKernelStackAndCalloutInternal+0x85
ffffdb80`b11556f0 fffff807`3f631e0e : ffff8507`4eb32c10 ffff8507`48e3ede0 ffffb389`00000000 ffff8507`48e3ede0 : NDIS!NdisSendNetBufferLists+0x559
ffffdb80`b1155820 fffff807`3f635502 : ffffb389`704b8701 00000000`00000001 ffffb389`704b8720 ffffb389`704b8700 : vmswitch!VmsExtPtRouteNetBufferLists+0x40e
ffffdb80`b11558f0 fffff802`94440001 : ffffdb80`b0e02f28 00000000`00000000 ffffdb80`b1155a60 00000000`00000000 : vmswitch!VmsPtHostRssStandardThreadedDpc+0x162
ffffdb80`b1155960 fffff802`94534823 : ffffdb80`b0e00180 00000000`00000080 ffffdb80`b0e02f6e ffffdb80`b0e00180 : nt!KiExecuteAllDpcs+0x2b1
ffffdb80`b1155ab0 fffff802`944a44bd : 00000000`00000000 ffffb389`6d7f1040 00000000`00000080 00000000`00000000 : nt!KiExecuteDpc+0x93
ffffdb80`b1155c10 fffff802`94557456 : ffffdb80`b0e00180 ffffb389`6d7f1040 fffff802`944a447c 00000000`00000000 : nt!PspSystemThreadStartup+0x41
ffffdb80`b1155c60 00000000`00000000 : ffffdb80`b1156000 ffffdb80`b1150000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16


STACK_COMMAND: .thread 0xffffb3896d7f1040 ; kb

THREAD_SHA1_HASH_MOD_FUNC: 764fccbe59568ce2929fd4238ae4ce1a75ef30c8

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 05b964782f1fa0304a5f80b5c7ff8eda7085d21b

THREAD_SHA1_HASH_MOD: eba13483c685b5460f071b095c39d72751763638

FOLLOWUP_IP:
vmswitch!VmsNblHelperCreateCloneNbl+c4
fffff807`3f626dd4 488bf8 mov rdi,rax

FAULT_INSTR_CODE: 48f88b48

SYMBOL_STACK_INDEX: c

SYMBOL_NAME: vmswitch!VmsNblHelperCreateCloneNbl+c4

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: vmswitch

IMAGE_NAME: vmswitch.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 578999dc

BUCKET_ID_FUNC_OFFSET: c4

FAILURE_BUCKET_ID: 0xB8_VRF_vmswitch!VmsNblHelperCreateCloneNbl

BUCKET_ID: 0xB8_VRF_vmswitch!VmsNblHelperCreateCloneNbl

PRIMARY_PROBLEM_CLASS: 0xB8_VRF_vmswitch!VmsNblHelperCreateCloneNbl

TARGET_TIME: 2016-08-28T17:31:05.000Z

OSBUILD: 14393

OSSERVICEPACK: 0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK: 402

PRODUCT_TYPE: 3

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

OSEDITION: Windows 10 Server Enterprise TerminalServer DataCenter SingleUserTS

OS_LOCALE:

USER_LCID: 0

OSBUILD_TIMESTAMP: 2016-07-15 19:16:17

BUILDDATESTAMP_STR: 160715-1616

BUILDLAB_STR: rs1_release

BUILDOSVER_STR: 10.0.14393.0.amd64fre.rs1_release.160715-1616

ANALYSIS_SESSION_ELAPSED_TIME: 4939

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:0xb8_vrf_vmswitch!vmsnblhelpercreateclonenbl

FAILURE_ID_HASH: {79380350-b249-c7d2-4a4b-b945082d85a0}

Followup: MachineOwner
---------

2: kd> .thread 0xffffb3896d7f1040
Implicit thread is now ffffb389`6d7f1040
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
Developing Minifilters 29 July 2019 OSR Seminar Space
Writing WDF Drivers 23 Sept 2019 OSR Seminar Space
Kernel Debugging 21 Oct 2019 OSR Seminar Space
Internals & Software Drivers 18 Nov 2019 Dulles, VA