Hung system - deadlock questions

NTDEV
1

I have a couple of questions regarding system hangs/deadlocks. The dump i am looking at was generated on a hung system ( Vista 64, 4 processor).

Output from the !locks command is as follows …

**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks.

Resource @ nt!PnpRegistryDeviceResource (0xfffff80003283260) Exclusively owned
Contention Count = 870
NumberOfExclusiveWaiters = 1
Threads: fffffa8003eb6b60-01<*>
Threads Waiting On Exclusive Access:
fffffa8008db7b60

KD: Scanning for held locks…

Resource @ fastfat!FatData (0xfffff88010b694f8) Exclusively owned
Contention Count = 1
NumberOfExclusiveWaiters = 1
Threads: fffffa8008313b60-01<*>
Threads Waiting On Exclusive Access:
fffffa8008882510

KD: Scanning for held locks…

Resource @ 0xfffffa80079d4790 Exclusively owned
Contention Count = 96987
NumberOfExclusiveWaiters = 10
Threads: fffffa8004057b60-01<*>
Threads Waiting On Exclusive Access:
fffffa800656db60 fffffa8006fba850 fffffa80091a5b60 fffffa800821a060
fffffa8003d19b60 fffffa8008e6e060 fffffa8008ddd060 fffffa8006adeb60
fffffa8003fbd060 fffffa800842cb60

KD: Scanning for held locks…
11619 total locks, 3 locks currently held

I had hoped that !locks would show a pattern whereby threadA holds resource1 and is waiting for exclusive access to resource2 which is owned by threadB that is waiting for exclusive access to resource1.(ie typical deadlock).

However given that “a deadlock is caused by one non-executing thread holding an exclusive lock on a resource that the executing thread needs” i thought i’d try to find out more about the excuting thread. However, when i do a !thread i get the stack trace below that looks suspiciously like its the thread that handles the operators request to force a crash.

THREAD fffff880009f4fc0 Cid 0000.0000 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 1
Not impersonating
DeviceMap fffff8a000008bb0
Owning Process fffffa8003c8d740 Image: System
Wait Start TickCount 0 Ticks: 26639 (0:00:06:55.571)
Context Switch Count 291675
UserTime 00:00:00.0000
KernelTime 00:06:25.0930
Win32 Start Address nt!KiIdleLoop (0xfffff80003087e90)
Stack Init fffff88002f1bd70 Current fffff88002f1bd00
Base fffff88002f1c000 Limit fffff88002f16000 Call 0
Priority 16 BasePriority 0 PriorityDecrement 0
Child-SP RetAddr : Args to Child : Call Site
fffff880009f4bb8 fffff800035fd894 : 0000000000000080 00000000004f4454 0000000000000000 0000000000000000 : nt!KeBugCheckEx
fffff880009f4bc0 fffff80003195593 : 0000000000000001 fffff800036132b0 0000000000000000 000000000000005c : hal!HalBugCheckSystem+0x160
fffff880009f4c00 fffff800035f76c1 : 00000000000006c0 fffff880009f4e20 fffff880009f4cf0 fffff800036132b0 : nt!WheaReportHwError+0x263
fffff880009f4c60 fffff80003139371 : fffff880009f4e30 0000000000000001 0000000000000001 0000000000000001 : hal!HalHandleNMI+0x149
fffff880009f4c90 fffff8000307ca42 : 0000000000000001 0000000000000000 0000000000000000 0000000000000001 : nt!KiProcessNMI+0x131
fffff880009f4cf0 fffff8000307c8a3 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KxNmiInterrupt+0x82
fffff880009f4e30 fffff880040ddc61 : fffff8000308d24a 00000000002824a4 fffffa8003ed6320 0000000000000000 : nt!KiNmiInterrupt+0x163 (TrapFrame @ fffff880009f4e30) fffff88002f1bc58 fffff8000308d24a : 00000000002824a4 fffffa8003ed6320 0000000000000000 0000000000000000 : intelppm+0x2c61 fffff88002f1bc60 fffff80003087ebc : fffff880009ea180 fffff88000000001 0000000000000001 fffff80000000000 : nt!PoIdle+0x53a fffff88002f1bd40 0000000000000000 : fffff88002f1c000 0000000000000000 0000000000000000 00000000`00000000 : nt!KiIdleLoop+0x2c

So, my questions are, under what circumstances would !locks not show a deadlock type pattern ? Also, is getting the currently executing thread on a hung system on which an operator has forced a crash a bit futile since it’ll be the one that bought the system down and not the one that was holding the lock that caused the deadlock?

thanks for any help/pointers/corrections-in-my-logic.

Paolo.

2

Sorry, !locks output got truncated …

**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks.

Resource @ nt!PnpRegistryDeviceResource (0xfffff80003283260) Exclusively owned
Contention Count = 870
NumberOfExclusiveWaiters = 1
Threads: fffffa8003eb6b60-01<*>
Threads Waiting On Exclusive Access:
fffffa8008db7b60

KD: Scanning for held locks…

Resource @ fastfat!FatData (0xfffff88010b694f8) Exclusively owned
Contention Count = 1
NumberOfExclusiveWaiters = 1
Threads: fffffa8008313b60-01<*>
Threads Waiting On Exclusive Access:
fffffa8008882510

KD: Scanning for held locks…

Resource @ 0xfffffa80079d4790 Exclusively owned
Contention Count = 96987
NumberOfExclusiveWaiters = 10
Threads: fffffa8004057b60-01<*>
Threads Waiting On Exclusive Access:
fffffa800656db60 fffffa8006fba850 fffffa80091a5b60 fffffa800821a060
fffffa8003d19b60 fffffa8008e6e060 fffffa8008ddd060 fffffa8006adeb60
fffffa8003fbd060 fffffa800842cb60

KD: Scanning for held locks…
11619 total locks, 3 locks currently held

3

!locks doesn’t track ALL locks… Just eresources. So, if your deadlock is due to some other kind of lock, !locks won’t help you.

Peter
OSR

4

Thanks for the info Peter. The main suspect in my code as far as deadlocks are concerned are KMUTEX’s which i believe are not ERESOURCE type locks ( have just read OSR’s “Synchronicity - A Review of Synchronization Primitives”).

Is there any simple way of finding out the KMUTEX type objects out there and which thread owns them ??

thanks,

Paolo

5

> Is there any simple way of finding out the KMUTEX type objects out there and which thread owns

them ??

!process 0 7 or some version of !stacks command to dump all stacks in system.

Sorry, there are no simpler ways. KMUTEX is not ERESOURCE.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

6

In fact, Paolo, at OSR we often find ourselves using ERESOURCES when just we need a mutual exclusion (not reader/writer) dispatcher-based synchronization mechanism (that is, something other than a spin lock), just BECAUSE !locks makes deadlocks so much easier to debug.

Using an ERESOURCE in place of a FAST_MUTEX or KGUARDED_MUTEX isn’t as performant, but unless you have something that’s in a critical path optimizing performance for such things might not be important… at least initially. Once you’re confident everything is working, you can always swap out the use of the ERESOURCE for a “more efficient” primitive – and this is particularly easy to do if you put the acquire/release in your own (force inline) function.

Hope that helps,

Peter
OSR