Hi,
I have a driver that share some traits with the nonpnp sample from the WDK. It’s a KMDF non-PnP driver that creates a single CDO.
It has a worker thread that’s created using PsCreateSystemThread(). Work items are queued into a regular doubly-linked list and the thread is woken using a notification event. This is working just as intended during normal operation.
There is also a lock used for the work item queue. The lock protects the wake-up event as well, to avoid any race conditions.
However, when attempting to unload the driver I find that the worker thread is not being scheduled as expected.
At the time of unloading, the worker is suspended inside KeWaitForSingleObject(), waiting for the wake-up event to be signaled:
kd> !thread 0xFFFFBC888D44C080
THREAD ffffbc888d44c080 Cid 0004.206c Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
Not impersonating
DeviceMap ffff910571c13ae0
Owning Process ffffbc8884662040 Image: System
Attached Process N/A Image: N/A
Wait Start TickCount 244327 Ticks: 1842018 (0:07:59:41.531)
Context Switch Count 20 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.015
Win32 Start Address mydriver!DispatchWorker (0xfffff8047363a420)
Stack Init ffff8c0fa265ac90 Current ffff8c0fa265a800
Base ffff8c0fa265b000 Limit ffff8c0fa2655000 Call 0000000000000000
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Args to Child : Call Site
ffff8c0f`a265a840 fffff804`740b968d : fffff804`00000001 00000000`fffffffe fffff804`00000008 00000000`00000002 : nt!KiSwapContext+0x76
ffff8c0f`a265a980 fffff804`740b8514 : ffffbc88`8d44c080 fffff804`741df215 00000000`00000010 00000000`00000000 : nt!KiSwapThread+0xbfd
ffff8c0f`a265aa20 fffff804`740b7cb5 : ffffbc88`84175e90 00000008`00000000 fffff804`75303900 00000000`00000000 : nt!KiCommitThreadWait+0x144
ffff8c0f`a265aac0 fffff804`7363a47a : ffff9105`762636c8 00000000`00000000 ffff9105`7f2bc100 00000000`00000000 : nt!KeWaitForSingleObject+0x255
ffff8c0f`a265aba0 fffff804`74133135 : ffff9105`76263690 fffff804`7363a420 ffff9105`76263690 ffff84d7`2cc84b81 : mydriver!DispatchWorker+0x5a [C:\work\mydriver\mydriver.cpp @ 151]
ffff8c0f`a265ac10 fffff804`741de9a8 : fffff804`73325180 ffffbc88`8d44c080 fffff804`741330e0 00000000`00000064 : nt!PspSystemThreadStartup+0x55
ffff8c0f`a265ac60 00000000`00000000 : ffff8c0f`a265b000 ffff8c0f`a2655000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x28
A system thread has called into the device object clean-up handler, and has just executed the following code:
WdfWaitLockAcquire(context->QueueLock, NULL);
KeSetEvent(&context->ExitWorker, 0, FALSE);
KeSetEvent(&context->WakeUpWorker, 0, FALSE);
WdfWaitLockRelease(context->QueueLock);
KeWaitForSingleObject(&context->DispatchWorker, Executive, KernelMode, FALSE, NULL);
Where DispatchWorker was acquired much earlier like so:
status = ObReferenceObjectByHandle
(
threadHandle,
THREAD_ALL_ACCESS,
NULL,
KernelMode,
(PVOID *)&context->DispatchWorker,
NULL
);
As the clean-up code starts waiting on the worker thread, the whole system dead-locks.
Here are the details for the system thread:
kd> !thread
THREAD ffffbc888474e040 Cid 0004.0058 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0
Not impersonating
DeviceMap ffff910571c13ae0
Owning Process ffffbc8884662040 Image: System
Attached Process N/A Image: N/A
Wait Start TickCount 261835 Ticks: 0
Context Switch Count 164696 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:08.484
Win32 Start Address nt!ExpWorkerThread (0xfffff80474098840)
Stack Init ffff8c0fa009ec90 Current ffff8c0fa009d7a0
Base ffff8c0fa009f000 Limit ffff8c0fa0099000 Call 0000000000000000
Priority 13 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Args to Child : Call Site
ffff8c0f`a009e910 fffff804`7535c457 : 00004377`74a463e8 ffff8c0f`a009e958 ffff8c0f`a009e958 64206c65`6e6e7501 : mydriver!CleanupDevice+0x70 [C:\work\mydriver\mydriver.cpp @ 197]
ffff8c0f`a009e950 fffff804`75353779 : ffffbc88`8b5b9c10 ffffbc88`8cba55b0 ffffbc88`8b5b9c30 fffff804`741e554c : Wdf01000!FxObject::CallCleanupCallbacks+0x4f [minkernel\wdf\framework\shared\object\fxobject.cpp @ 354]
(Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : Wdf01000!FxObject::CallCleanup+0x16 (Inline Function @ fffff804`75353779) [minkernel\wdf\framework\shared\inc\private\common\fxobject.hpp @ 831]
ffff8c0f`a009e980 fffff804`75326c5b : ffffbc88`8b5b9c10 ffffbc88`8cba55b0 ffffbc88`8bf5bb08 00001f80`01083a55 : Wdf01000!FxDevice::Dispose+0x39 [minkernel\wdf\framework\shared\core\fxdevice.cpp @ 1306]
ffff8c0f`a009e9b0 fffff804`7530821d : ffffbc88`8cba55b0 ffffbc88`8a69fb60 ffffbc88`8b5b9c58 00000000`00000000 : Wdf01000!FxObject::DisposeChildrenWorker+0x1e5bb [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 1212]
(Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : Wdf01000!FxObject::PerformDisposingDisposeChildrenLocked+0x25 (Inline Function @ fffff804`7530821d) [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 846]
(Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : Wdf01000!FxObject::PerformEarlyDisposeWorkerAndUnlock+0x43 (Inline Function @ fffff804`7530821d) [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 924]
(Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : Wdf01000!FxObject::PerformEarlyDispose+0x65 (Inline Function @ fffff804`7530821d) [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 553]
(Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : Wdf01000!FxObject::DisposeChildrenWorker+0x161 (Inline Function @ fffff804`7530821d) [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 1191]
(Inline Function) --------`-------- : --------`-------- --------`-------- --------`-------- --------`-------- : Wdf01000!FxObject::DeleteWorkerAndUnlock+0x1a0 (Inline Function @ fffff804`7530821d) [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 968]
ffff8c0f`a009e9f0 fffff804`753555d9 : ffffbc88`8bf90e00 00004377`75960498 00000000`00000000 00000000`00000001 : Wdf01000!FxObject::DeleteObject+0x26d [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 124]
ffff8c0f`a009ea70 fffff804`753559c8 : 00004377`75960498 00000000`00000000 00000000`00000080 ffffbc88`846acca0 : Wdf01000!FxDriver::DeleteObject+0x9 [minkernel\wdf\framework\shared\inc\private\common\fxdriver.hpp @ 375]
ffff8c0f`a009eaa0 fffff804`7363b82e : ffffbc88`8474e040 fffff804`7470d9c0 ffffbc88`846acca0 00000000`00010282 : Wdf01000!FxDriver::Unload+0xe8 [minkernel\wdf\framework\shared\core\fxdriver.cpp @ 201]
ffff8c0f`a009eb00 fffff804`747fbb13 : 00000000`00000000 00000000`00000000 00000000`00000300 fffff804`7447aa60 : mydriver!FxStubDriverUnload+0x1e [d:\w8rtm\minkernel\wdf\framework\kmdf\src\dynamic\stub\stub.cpp @ 158]
ffff8c0f`a009eb30 fffff804`74098945 : ffffbc88`846acca0 fffff804`7462f900 ffffbc88`8474e040 ffffbc88`0000000c : nt!IopLoadUnloadDriver+0xee153
ffff8c0f`a009eb70 fffff804`74133135 : ffffbc88`8474e040 00000000`00000080 ffffbc88`84662040 00000000`00000000 : nt!ExpWorkerThread+0x105
ffff8c0f`a009ec10 fffff804`741de9a8 : fffff804`73325180 ffffbc88`8474e040 fffff804`741330e0 00000000`00000000 : nt!PspSystemThreadStartup+0x55
ffff8c0f`a009ec60 00000000`00000000 : ffff8c0f`a009f000 ffff8c0f`a0099000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x28
The only thing I notice that seems a little off is the raised priority of the system thread.
Is this a case of “don’t use KeWaitForSingleObject() in an arbitrary context”? If so, what is a better design to solve this?
TIA