0x9E on Exchange

Malcolm_McCaffery · October 26, 2015, 7:29pm

Wondering if there is any indication from this on what user mode
components to investigate relating to the hang, or any best “next
steps” There is only a kernel dmp file, should I be going for a
complete dmp or are there some clues here as to possible offenders.
I was advised machine had “increased disk latency” before hang, but
didn’t seem excessively so. In the meantime I am aiming to get PerfMon
running with PAL Tools Exchange profile.

VIRTUAL_MACHINE: VMware
SYSTEM_VERSION: None
BIOS_DATE: 06/11/2014
BASEBOARD_PRODUCT: 440BX Desktop Reference Platform
BASEBOARD_VERSION: None
BUGCHECK_P1: fffffa803494ab10
BUGCHECK_P2: 3c
BUGCHECK_P3: 0
BUGCHECK_P4: 0
PROCESS_NAME: clussvc.exe
IMAGE_NAME: clussvc.exe
DEBUG_FLR_IMAGE_TIMESTAMP: 0
MODULE_NAME: clussvc
FAULTING_MODULE: 0000000000000000
CPU_COUNT: c
CPU_MHZ: 7cb
CPU_VENDOR: GenuineIntel
CPU_FAMILY: 6
CPU_MODEL: f
CPU_STEPPING: 1
DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT
BUGCHECK_STR: 0x9E
CURRENT_IRQL: 2
ANALYSIS_VERSION: 10.0.10166.9 amd64fre
DPC_STACK_BASE: FFFFF880024B0FB0
11: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 12582782 ( 50331128 Kb)
Page File: ??\D:\pagefile.sys
Current: 50341888 Kb Free Space: 50341884 Kb
Minimum: 50341888 Kb Maximum: 50341888 Kb
Available Pages: 2255783 ( 9023132 Kb)
ResAvail Pages: 11991193 ( 47964772 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 33503357 ( 134013428 Kb)
Modified Pages: 28044 ( 112176 Kb)
Modified PF Pages: 28039 ( 112156 Kb)
Modified No Write Pages: 0 ( 0 Kb)
NonPagedPool 0 Used: 7932 ( 31728 Kb)
NonPagedPool 1 Used: 11211 ( 44844 Kb)
NonPagedPool 2 Used: 86901 ( 347604 Kb)
NonPagedPool 3 Used: 17211 ( 68844 Kb)
NonPagedPool 4 Used: 10356 ( 41424 Kb)
NonPagedPool 5 Used: 13283 ( 53132 Kb)
NonPagedPool Usage: 159335 ( 637340 Kb)
NonPagedPool Max: 9407989 ( 37631956 Kb)
PagedPool 0 Usage: 110802 ( 443208 Kb)
PagedPool 1 Usage: 30340 ( 121360 Kb)
PagedPool 2 Usage: 28208 ( 112832 Kb)
PagedPool 3 Usage: 289560 ( 1158240 Kb)
PagedPool 4 Usage: 12138 ( 48552 Kb)
PagedPool 5 Usage: 29155 ( 116620 Kb)
PagedPool 6 Usage: 35496 ( 141984 Kb)
PagedPool Usage: 535699 ( 2142796 Kb)
PagedPool Maximum: 33554432 ( 134217728 Kb)
Session Commit: 3337 ( 13348 Kb)
Shared Commit: 197060 ( 788240 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 19209 ( 76836 Kb)
Pages For MDLs: 77 ( 308 Kb)
Pages For AWE: 0 ( 0 Kb)
NonPagedPool Commit: 0 ( 0 Kb)
PagedPool Commit: 536006 ( 2144024 Kb)
Driver Commit: 4594 ( 18376 Kb)
Boot Commit: 0 ( 0 Kb)
System PageTables: 0 ( 0 Kb)
VAD/PageTable Bitmaps: 6703 ( 26812 Kb)
ProcessLockedFilePages: 0 ( 0 Kb)
Pagefile Hash Pages: 0 ( 0 Kb)
Sum System Commit: 766986 ( 3067944 Kb)
Total Private: 10919669 ( 43678676 Kb)
Misc/Transient Commit: 569539 ( 2278156 Kb)
Committed pages: 12256194 ( 49024776 Kb)
Commit limit: 25167790 ( 100671160 Kb)

Pid ImageName Commit(P) Commit(KB) Debt(P) Debt(KB)

1818 store.exe 7967305 ( 31869220 Kb) 0 ( 0 Kb)
0818 Microsoft.Exch 409445 ( 1637780 Kb) 0 ( 0 Kb)
1378 EdgeTransport. 260942 ( 1043768 Kb) 0 ( 0 Kb)
0f14 Microsoft.Exch 258830 ( 1035320 Kb) 0 ( 0 Kb)
43c0 w3wp.exe 222036 ( 888144 Kb) 0 ( 0 Kb)
0b94 Microsoft.Exch 142448 ( 569792 Kb) 0 ( 0 Kb)
12cc MSExchangeTran 131908 ( 527632 Kb) 0 ( 0 Kb)
0c20 MSExchangeMail 116451 ( 465804 Kb) 0 ( 0 Kb)
199c w3wp.exe 104518 ( 418072 Kb) 0 ( 0 Kb)
0df8 Microsoft.Exch 91708 ( 366832 Kb) 0 ( 0 Kb)
0ea0 msexchangerepl 78011 ( 312044 Kb) 0 ( 0 Kb)
2bb0 msftefd.exe 73476 ( 293904 Kb) 0 ( 0 Kb)
36e8 msftefd.exe 72888 ( 291552 Kb) 0 ( 0 Kb)
1ce8 msftefd.exe 67573 ( 270292 Kb) 0 ( 0 Kb)
3570 w3wp.exe 66111 ( 264444 Kb) 0 ( 0 Kb)
0fc4 Microsoft.Exch 60633 ( 242532 Kb) 0 ( 0 Kb)
0f7c Microsoft.Exch 52015 ( 208060 Kb) 0 ( 0 Kb)
2364 w3wp.exe 43589 ( 174356 Kb) 0 ( 0 Kb)
0ab4 Microsoft.Exch 40492 ( 161968 Kb) 0 ( 0 Kb)
3058 w3wp.exe 40277 ( 161108 Kb) 0 ( 0 Kb)
1f48 w3wp.exe 39067 ( 156268 Kb) 0 ( 0 Kb)
0d1c Microsoft.Exch 38789 ( 155156 Kb) 0 ( 0 Kb)
09d4 Microsoft.Exch 36029 ( 144116 Kb) 0 ( 0 Kb)
10c4 MSExchangeThro 34814 ( 139256 Kb) 0 ( 0 Kb)
0db0 Microsoft.Exch 34266 ( 137064 Kb) 0 ( 0 Kb)
0ce4 powershell.exe 33479 ( 133916 Kb) 0 ( 0 Kb)
2f44 msftefd.exe 31927 ( 127708 Kb) 0 ( 0 Kb)
2ae4 msftefd.exe 27341 ( 109364 Kb) 0 ( 0 Kb)
0b74 MSExchangeMail 27260 ( 109040 Kb) 0 ( 0 Kb)
4284 msftefd.exe 26387 ( 105548 Kb) 0 ( 0 Kb)
0cc0 MSExchangeMail 24007 ( 96028 Kb) 0 ( 0 Kb)
15e8 bpinetd.exe 23888 ( 95552 Kb) 0 ( 0 Kb)
0a5c MsExchangeFDS. 20520 ( 82080 Kb) 0 ( 0 Kb)
11e8 MSExchangeTran 18606 ( 74424 Kb) 0 ( 0 Kb)
02e4 lsass.exe 13634 ( 54536 Kb) 0 ( 0 Kb)
0998 Microsoft.Exch 11125 ( 44500 Kb) 0 ( 0 Kb)
0630 rundll32.exe 9288 ( 37152 Kb) 0 ( 0 Kb)
18cc mad.exe 7822 ( 31288 Kb) 0 ( 0 Kb)
07dc ccSvcHst.exe 7745 ( 30980 Kb) 0 ( 0 Kb)
0748 SMSvcHost.exe 7455 ( 29820 Kb) 0 ( 0 Kb)
05e4 inetinfo.exe 7340 ( 29360 Kb) 0 ( 0 Kb)
2ec4 msftefd.exe 6821 ( 27284 Kb) 0 ( 0 Kb)
36e0 perfmon.exe 6648 ( 26592 Kb) 0 ( 0 Kb)
1360 nbdisco.exe 6619 ( 26476 Kb) 0 ( 0 Kb)
1bf0 Smc.exe 6569 ( 26276 Kb) 0 ( 0 Kb)
00a8 svchost.exe 6540 ( 26160 Kb) 0 ( 0 Kb)
02e8 svchost.exe 5979 ( 23916 Kb) 0 ( 0 Kb)
07c8 svchost.exe 5924 ( 23696 Kb) 0 ( 0 Kb)
0690 MSExchangeADTo 5755 ( 23020 Kb) 0 ( 0 Kb)
00b0 svchost.exe 4881 ( 19524 Kb) 0 ( 0 Kb)
06cc exfba.exe 4399 ( 17596 Kb) 0 ( 0 Kb)
06f4 msftesql.exe 4380 ( 17520 Kb) 0 ( 0 Kb)
2308 bpfis.exe 3974 ( 15896 Kb) 0 ( 0 Kb)
166c clussvc.exe 2977 ( 11908 Kb) 0 ( 0 Kb)
067c svchost.exe 2865 ( 11460 Kb) 0 ( 0 Kb)
0460 svchost.exe 2807 ( 11228 Kb) 0 ( 0 Kb)
021c svchost.exe 2603 ( 10412 Kb) 0 ( 0 Kb)
02dc services.exe 2491 ( 9964 Kb) 0 ( 0 Kb)
04fc spoolsv.exe 2466 ( 9864 Kb) 0 ( 0 Kb)
092c processes.exe 2454 ( 9816 Kb) 0 ( 0 Kb)
0160 LogonUI.exe 2400 ( 9600 Kb) 0 ( 0 Kb)
0a6c cdm.exe 2305 ( 9220 Kb) 0 ( 0 Kb)
15b0 vmtoolsd.exe 2278 ( 9112 Kb) 0 ( 0 Kb)
0274 csrss.exe 2209 ( 8836 Kb) 0 ( 0 Kb)
1b04 ntevl.exe 2054 ( 8216 Kb) 0 ( 0 Kb)
03a4 svchost.exe 2036 ( 8144 Kb) 0 ( 0 Kb)
09b0 exchange_monit 1984 ( 7936 Kb) 0 ( 0 Kb)
1ba8 svchost.exe 1980 ( 7920 Kb) 0 ( 0 Kb)
0350 svchost.exe 1742 ( 6968 Kb) 0 ( 0 Kb)
16b8 bpcd.exe 1721 ( 6884 Kb) 0 ( 0 Kb)
05c8 svchost.exe 1517 ( 6068 Kb) 0 ( 0 Kb)
052c svchost.exe 1456 ( 5824 Kb) 0 ( 0 Kb)
1b6c rhs.exe 1429 ( 5716 Kb) 0 ( 0 Kb)
07e4 controller.exe 1395 ( 5580 Kb) 0 ( 0 Kb)
1d04 dllhost.exe 1309 ( 5236 Kb) 0 ( 0 Kb)
0240 csrss.exe 1177 ( 4708 Kb) 0 ( 0 Kb)
1cb4 svchost.exe 1153 ( 4612 Kb) 0 ( 0 Kb)
0728 mtstrmd.exe 1150 ( 4600 Kb) 0 ( 0 Kb)
1e40 msdtc.exe 1130 ( 4520 Kb) 0 ( 0 Kb)
0b4c ntservices.exe 1093 ( 4372 Kb) 0 ( 0 Kb)
1524 vnetd.exe 1091 ( 4364 Kb) 0 ( 0 Kb)
08d4 spooler.exe 1036 ( 4144 Kb) 0 ( 0 Kb)
02ec lsm.exe 980 ( 3920 Kb) 0 ( 0 Kb)
1ab8 rhs.exe 957 ( 3828 Kb) 0 ( 0 Kb)
0494 pbx_exchange.e 897 ( 3588 Kb) 0 ( 0 Kb)
05c0 taskeng.exe 880 ( 3520 Kb) 0 ( 0 Kb)
1da0 VSSVC.exe 876 ( 3504 Kb) 0 ( 0 Kb)
44c0 OleConverter.e 875 ( 3500 Kb) 0 ( 0 Kb)
0914 hdb.exe 838 ( 3352 Kb) 0 ( 0 Kb)
0798 nimbus.exe 746 ( 2984 Kb) 0 ( 0 Kb)
02a0 winlogon.exe 735 ( 2940 Kb) 0 ( 0 Kb)
0254 svchost.exe 718 ( 2872 Kb) 0 ( 0 Kb)
1ce0 svchost.exe 709 ( 2836 Kb) 0 ( 0 Kb)
24a0 svchost.exe 655 ( 2620 Kb) 0 ( 0 Kb)
4b10 starter.exe 650 ( 2600 Kb) 0 ( 0 Kb)
027c wininit.exe 631 ( 2524 Kb) 0 ( 0 Kb)
0698 lic98Service.e 554 ( 2216 Kb) 0 ( 0 Kb)
3060 conhost.exe 489 ( 1956 Kb) 0 ( 0 Kb)
07a4 conhost.exe 489 ( 1956 Kb) 0 ( 0 Kb)
1384 conhost.exe 488 ( 1952 Kb) 0 ( 0 Kb)
0bac conhost.exe 488 ( 1952 Kb) 0 ( 0 Kb)
2314 conhost.exe 487 ( 1948 Kb) 0 ( 0 Kb)
0e04 conhost.exe 487 ( 1948 Kb) 0 ( 0 Kb)
061c LogWatNT.exe 380 ( 1520 Kb) 0 ( 0 Kb)
01a0 smss.exe 274 ( 1096 Kb) 0 ( 0 Kb)
0004 System 144 ( 576 Kb) 0 ( 0 Kb)
4ab8 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
4a40 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
49d4 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
4938 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
35ac appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
3364 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
2f3c appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
2cac appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
2a58 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
259c appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
235c appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
22f8 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
1fbc w3wp.exe 0 ( 0 Kb) 0 ( 0 Kb)
1f84 w3wp.exe 0 ( 0 Kb) 0 ( 0 Kb)
1e34 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
13cc appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
1324 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
1250 appcmd.exe 0 ( 0 Kb) 0 ( 0 Kb)
11: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks.

Resource @ nt!CmpRegistryLock (0xfffff80002050000) Shared 12 owning threads
Contention Count = 15
Threads: fffffa80715d7060-01<*> fffffa802512d5f0-01<*>
fffffa803956bb50-01<*> fffffa8025166b50-01<*>
fffffa80396065c0-01<*> fffffa8052f83600-01<*> fffffa8065e50060-01<*>
fffffa8025fc8060-01<*>
fffffa80275e8b50-01<*> fffffa8052ea7940-01<*> fffffa80348d6060-01<*>
fffffa803a14eb50-01<*>
KD: Scanning for held locks

Resource @ 0xfffffa8071209888 Shared 23 owning threads
Contention Count = 87
Threads: fffffa8043c62b50-01<*> fffffa8038cfc650-01<*>
fffffa803ab53060-01<*> fffffa80714ebad0-01<*>
fffffa8062386590-01<*> fffffa805da91b50-01<*> fffffa80269bdb50-01<*>
fffffa806300b060-01<*>
fffffa80621a5b50-01<*> fffffa8061fb6060-01<*> fffffa80715d7640-01<*>
fffffa80715a6060-01<*>
fffffa8036650b50-01<*> fffffa80715768b0-01<*> fffffa80624ef5d0-01<*>
fffffa803a132060-01<*>
fffffa8043d50060-01<*> fffffa8062bc7670-01<*> fffffa80347a8a00-01<*>
fffffa8052d9d610-01<*>
fffffa8062492830-01<*> fffffa8038bf1060-01<*> fffffa802810db50-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa807120a368 Shared 21 owning threads
Contention Count = 338
Threads: fffffa8038cfc650-01<*> fffffa8043c62b50-01<*>
fffffa803ab53060-01<*> fffffa80714ebad0-01<*>
fffffa8062386590-01<*> fffffa805da91b50-01<*> fffffa806300b060-01<*>
fffffa80621a5b50-01<*>
fffffa8061fb6060-01<*> fffffa80715a6060-01<*> fffffa80715d7640-01<*>
fffffa8036650b50-01<*>
fffffa80715768b0-01<*> fffffa80624ef5d0-01<*> fffffa803a132060-01<*>
fffffa8043d50060-01<*>
fffffa8062bc7670-01<*> fffffa80347a8a00-01<*> fffffa8052d9d610-01<*>
fffffa8062492830-01<*>
fffffa802810db50-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa802575ff50 Exclusively owned
Contention Count = 173
Threads: fffffa807162e060-01<*>
KD: Scanning for held locks.

*** ERROR: Module load completed but symbols could not be loaded for SRTSP64.SYS
Resource @ SRTSP64 (0xfffff88002f38dd0) Shared 2 owning threads
Threads: fffffa8043ff8b50-01<*> fffffa8025a64040-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8025a25f90 Shared 1 owning threads
Threads: fffffa8063023a00-01<*>

Resource @ 0xfffffa8025a25f10 Shared 1 owning threads
Threads: fffffa8035975060-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa807137a888 Shared 1 owning threads
Contention Count = 27
Threads: fffffa80597d3b50-01<*>

Resource @ 0xfffffa807137a8f0 Shared 1 owning threads
Threads: fffffa80597d3b50-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8025b3e888 Shared 1 owning threads
Contention Count = 20
Threads: fffffa805aab62a0-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8025b45590 Exclusively owned
Contention Count = 3780
Threads: fffffa805aab62a0-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8052b13888 Shared 1 owning threads
Contention Count = 10
Threads: fffffa8071cf1b50-01<*>
KD: Scanning for held locks

Resource @ 0xfffffa8025f53888 Shared 1 owning threads
Contention Count = 7
Threads: fffffa803a187640-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8025f53528 Exclusively owned
Contention Count = 288
Threads: fffffa803a187640-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8052b94960 Shared 1 owning threads
Contention Count = 3131
Threads: fffffa8043f1a9b0-01<*>
KD: Scanning for held locks

Resource @ 0xfffffa80259eb438 Shared 1 owning threads
Contention Count = 158
Threads: fffffa80396065c0-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8052a74a58 Shared 1 owning threads
Threads: fffffa8025181b50-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa80345e7ab8 Shared 1 owning threads
Contention Count = 18
Threads: fffffa80385c6b50-01<*>
KD: Scanning for held locks

Resource @ 0xfffffa80347afd00 Shared 1 owning threads
Threads: fffffa805aab62a0-01<*>
KD: Scanning for held locks

Resource @ 0xfffffa8043cbfed0 Exclusively owned
Contention Count = 69
Threads: fffffa8071cf1b50-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8043d52830 Shared 1 owning threads
Contention Count = 24930
Threads: fffffa80280b0230-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8043dd6050 Exclusively owned
Contention Count = 8
Threads: fffffa807162e060-01<*>

Resource @ 0xfffffa8043dd60f8 Shared 1 owning threads
Contention Count = 10
Threads: fffffa807162e060-01<*>
KD: Scanning for held locks

Resource @ 0xfffffa8043fe59f0 Shared 1 owning threads
Threads: fffffa8038bf1060-01<*>
KD: Scanning for held locks.
Resource @ 0xfffffa8039ce2660 Exclusively owned
Threads: fffffa803a187640-01<*>

Resource @ 0xfffffa8039ce2708 Exclusively owned
Threads: fffffa803a187640-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8039f61f98 Shared 1 owning threads
Threads: fffffa80558a9940-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa803a121b38 Shared 1 owning threads
Threads: fffffa8038fafa90-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa805c187a48 Exclusively owned
Threads: fffffa80597d3b50-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa80591a7a38 Shared 1 owning threads
Threads: fffffa803a187640-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8049f72058 Shared 1 owning threads
Threads: fffffa8071cf1b50-01<*>

Resource @ 0xfffff8800bcc8790 Exclusively owned
Threads: fffffa80597d3b50-01<*>
KD: Scanning for held locks.

Resource @ 0xfffffa8072d4f3c8 Shared 1 owning threads
Threads: fffffa807162e060-01<*>
735448 total locks, 33 locks currently held
11: kd> !exqueue /f
**** Critical WorkQueue ( Threads: 15/512, Concurrency: 4/12 )
THREAD fffffa8024f51850 Cid 0004.001c Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa8024f51360 Cid 0004.0020 Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 7
THREAD fffffa8024f50040 Cid 0004.0024 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa8024f50b50 Cid 0004.0028 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa8024f50660 Cid 0004.002c Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa8024f4f040 Cid 0004.0030 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa8024f4fb50 Cid 0004.0034 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa8024f4f660 Cid 0004.0038 Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 7
THREAD fffffa8024f4e040 Cid 0004.003c Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa8024f4eb50 Cid 0004.0040 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa80436d8040 Cid 0004.01cc Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 7
THREAD fffffa80437c4040 Cid 0004.0204 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa805a611a50 Cid 0004.35f4 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa80654dc040 Cid 0004.3030 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa80597d3b50 Cid 0004.30f0 Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 6

**** Delayed WorkQueue ( Threads: 12/512, Concurrency: 7/12 )
THREAD fffffa8024f4e660 Cid 0004.0044 Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 7
THREAD fffffa8024f4d040 Cid 0004.0048 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa8024f4db50 Cid 0004.004c Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 3
THREAD fffffa8024f4d660 Cid 0004.0050 Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 7
THREAD fffffa8024f4c040 Cid 0004.0054 Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 7
THREAD fffffa8024f4c660 Cid 0004.005c Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 7
THREAD fffffa8024f4cb50 Cid 0004.0058 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa803966f820 Cid 0004.3538 Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 6
THREAD fffffa8051963380 Cid 0004.2450 Teb: 0000000000000000
Win32Thread: 0000000000000000 READY on processor 6
THREAD fffffa8059fe8a00 Cid 0004.3d80 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa80397c1b50 Cid 0004.2d20 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT
THREAD fffffa803aaa3b50 Cid 0004.3fb8 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT

**** HyperCritical WorkQueue ( Threads: 1/512, Concurrency: 0/12 )
THREAD fffffa8024f61040 Cid 0004.0060 Teb: 0000000000000000
Win32Thread: 0000000000000000 WAIT

11: kd> !running -it

System Processors: (0000000000000fff)
Idle Processors: (0000000000000000) (0000000000000000)
(0000000000000000) (0000000000000000)

Prcbs Current (pri) Next (pri) Idle
0 fffff80002046e80 fffffa8072d40b50 ( 8) fffff80002054cc0

0k

Child-SP RetAddr Call Site

00 000000000e1fdfe8 0000000000000000 0x000007fe`d7fc5ae0

1 fffff88002000180 fffffa80396a3b50 ( 9) fffff8800200b1c0

1k

Child-SP RetAddr Call Site

00 fffff88008943ce0 fffff80002193bf5 nt!ExAllocatePoolWithTag+0x517
01 fffff88008943dd0 fffff8000219a102 nt!AlpcpAllocateBlob+0x35
02 fffff88008943e00 fffff80002199eea nt!AlpcpCreateSecurityContext+0x42
03 fffff88008943ea0 fffff80002199c99
nt!AlpcpCaptureSecurityAttributeInternal+0xc2
04 fffff88008943ee0 fffff8000218aacf nt!AlpcpCaptureSecurityAttribute+0xc9
05 fffff88008943f40 fffff800021d521f nt!AlpcpCaptureAttributes+0x33c
06 fffff88008943fa0 fffff800021d7c5b nt!AlpcpSendMessage+0x6c5
07 fffff880089440b0 fffff800021d5a9b nt!AlpcpProcessSynchronousRequest+0x2f8
08 fffff88008944200 fffff80001ec7553 nt!NtAlpcSendWaitReceivePort+0x1ab
09 fffff880089442b0 fffff80001ec3b10 nt!KiSystemServiceCopyEnd+0x13
0a fffff880089444b8 fffff88000e1939b nt!KiServiceLinkage
0b fffff880089444c0 fffff88000e024e3 msrpc!LRPC_BASE_CCALL::SendReceive+0x19b
0c fffff88008944530 fffff88000e02c45 msrpc!NdrpClientCall3+0x4d3
0d fffff880089447f0 fffff880017be522 msrpc!NdrClientCall3+0xf6
0e fffff88008944b80 fffff880017cf304 ksecdd!SspipDeleteSecurityContext+0x1c2
0f fffff88008944c00 fffff88005932390 ksecdd!DeleteSecurityContext+0xa4
10 fffff88008944c40 fffff880058d68f4 HTTP!UxSslFreeConnectionControl+0x90
11 fffff88008944c70 fffff88005925289 HTTP!UxTlDestroyConnectionWorker+0x104
12 fffff88008944ca0 fffff8000215ff26 HTTP!UlpThreadPoolWorker+0x279
13 fffff88008944d40 fffff80001eb9646 nt!PspSystemThreadStartup+0x5a
14 fffff88008944d80 0000000000000000 nt!KxStartSystemThread+0x16

2 fffff88002071180 fffffa80399c8b50 (15) fffff8800207c1c0

2k

Child-SP RetAddr Call Site

00 000000001b82f560 0000000000000000 0x7761fa1f

3 fffff880020e2180 fffffa8071b70b50 (15) fffff880020ed1c0

3k
*** ERROR: Module load completed but symbols could not be loaded for
BHDrvx64.sys

Child-SP RetAddr Call Site

00 fffff880096b4430 fffff88003d67bf8 BHDrvx64+0x11dd7e
01 fffff880096b4470 fffff88003d67b06 BHDrvx64+0x11dbf8
02 fffff880096b44b0 fffff88003d146e2 BHDrvx64+0x11db06
03 fffff880096b4520 fffff88003cf97fe BHDrvx64+0xca6e2
04 fffff880096b45e0 fffff88003c527b2 BHDrvx64+0xaf7fe
05 fffff880096b4630 fffff88003c5231c BHDrvx64+0x87b2
06 fffff880096b4760 fffff88003cee10c BHDrvx64+0x831c
07 fffff880096b47a0 fffff88003cee096 BHDrvx64+0xa410c
08 fffff880096b47d0 fffff88003cecf63 BHDrvx64+0xa4096
09 fffff880096b4830 fffff88003cef20a BHDrvx64+0xa2f63
0a fffff880096b4920 fffff88003ceffd8 BHDrvx64+0xa520a
0b fffff880096b4970 fffff800022794d0 BHDrvx64+0xa5fd8
0c fffff880096b49b0 fffff8000222cc06 nt!CmpCallCallBacks+0x1c0
0d fffff880096b4a80 fffff80001ec7553 nt! ?? ::NNGAKEGL::string'+0x41432 0e fffff880096b4bb0 000000007764c43a nt!KiSystemServiceCopyEnd+0x13 0f 0000000067d1da48 00000000`00000000 0x7764c43a

4 fffff88002153180 fffffa8071699580 ( 8) fffff8800215e1c0

4k

Child-SP RetAddr Call Site

00 fffff880021817d0 fffff80001fffb0d nt!KxWaitForLockOwnerShip+0x30
*** ERROR: Module load completed but symbols could not be loaded for SYMNETS.SYS
01 fffff88002181800 fffff88003a1ba33 nt!ExAllocatePoolWithTag+0xa1d
02 fffff880021818f0 fffff88003a20121 SYMNETS+0x1ba33
03 fffff88002181920 fffff88003a1c34f SYMNETS+0x20121
04 fffff88002181970 fffff88000e6a782 SYMNETS+0x1c34f
05 fffff880021819f0 fffff88000e62f58 NETIO!ProcessCallout+0x1a2
06 fffff88002181b10 fffff88000e645d2 NETIO!ArbitrateAndEnforce+0x238
07 fffff88002181be0 fffff88001aa8d82 NETIO!KfdClassify+0x934
08 fffff88002181f50 fffff88001a7d444 tcpip! ?? ::FNODOBFM::string'+0x13992 09 fffff880021823a0 fffff88001a7d9e4 tcpip!WfpAleEndpointDeactivationHandler+0x34 0a fffff880021823f0 fffff88001a7e1f6 tcpip!TcpRemoveTcb+0x84 0b fffff88002182440 fffff88001a4a8e0 tcpip!TcpShutdownTcb+0x3e6 0c fffff88002182580 fffff88001b3581e tcpip!TcpAbortTcbDelivery+0x30 0d fffff880021825b0 fffff88001ac394e tcpip!TcpKeepAliveTimeout+0x1be 0e fffff88002182650 fffff88001a8b37d tcpip! ?? ::FNODOBFM::string’+0x333b9
0f fffff88002182720 fffff80001ed34cc tcpip!TcpPeriodicTimeoutHandler+0x265
10 fffff880021827a0 fffff80001ed3366 nt!KiProcessTimerDpcTable+0x6c
11 fffff88002182810 fffff80001ed324e nt!KiProcessExpiredTimerList+0xc6
12 fffff88002182e60 fffff80001ed3037 nt!KiTimerExpiration+0x1be
13 fffff88002182f00 fffff80001ecb415 nt!KiRetireDpcList+0x277
14 fffff88002182fb0 fffff80001ecb22c nt!KyRetireDpcList+0x5
15 fffff880077bbbe0 fffff80001f13253 nt!KiDispatchInterruptContinue
16 fffff880077bbc10 fffff80001ed36d8 nt!KiDpcInterruptBypass+0x13
17 fffff880077bbc20 000007fefd4719d6 nt!KiSecondaryClockInterrupt+0x1a8
18 000000006a4de000 0000000000000000 0x000007fe`fd4719d6

5 fffff880021c4180 fffffa8059865b50 ( 8) fffff880021cf1c0

5k

Child-SP RetAddr Call Site

00 000000002037f1a8 0000000000000000 0x000007fe`f8ff4826

6 fffff88002240180 fffffa8043e8bb50 (13) fffff8800224b1c0

6k

Child-SP RetAddr Call Site

00 fffff88008308af0 fffff800021af9e9 nt!ObReferenceObjectByHandleWithTag+0x22a
01 fffff88008308bc0 fffff80001ec7553 nt!NtSetEvent+0x75
02 fffff88008308c20 000000007764bf1a nt!KiSystemServiceCopyEnd+0x13
03 00000000024ff8a8 0000000000000000 0x7764bf1a

7 fffff880022b1180 fffffa8043fe1060 (15) fffff880022bc1c0

7k

Child-SP RetAddr Call Site

00 fffff880022df720 fffff800020001b1 nt!ExDeferredFreePool+0x20b
01 fffff880022df7b0 fffff88003654bec nt!ExFreePoolWithTag+0x411
02 fffff880022df860 fffff880036426dc afd! ?? ::GFJBLGFE::string'+0x4714 03 fffff880022df8f0 fffff88001a4a94b afd!AfdTLDisconnectEventHandler+0x4c 04 fffff880022df930 fffff88001a7e2c9 tcpip!TcpNotifyAbortDelivery+0x4b 05 fffff880022df960 fffff88001a4a8e0 tcpip!TcpShutdownTcb+0x4b9 06 fffff880022dfaa0 fffff88001a73d4b tcpip!TcpAbortTcbDelivery+0x30 07 fffff880022dfad0 fffff88001a61a47 tcpip!TcpTcbCarefulDatagram+0x32b 08 fffff880022dfc80 fffff88001a5efda tcpip!TcpTcbReceive+0x3ab 09 fffff880022dfd90 fffff88001a5fe9c tcpip!TcpMatchReceive+0x1fa 0a fffff880022dfee0 fffff88001a58137 tcpip!TcpPreValidatedReceive+0x49c 0b fffff880022dffb0 fffff88001a57caa tcpip!IppDeliverListToProtocol+0x97 0c fffff880022e0070 fffff88001a57261 tcpip!IppProcessDeliverList+0x5a 0d fffff880022e0110 fffff88001a54eef tcpip!IppReceiveHeaderBatch+0x232 0e fffff880022e0210 fffff88001a544c2 tcpip!IpFlcReceivePackets+0x64f 0f fffff880022e0410 fffff88001a538ea tcpip!FlpReceiveNonPreValidatedNetBufferListChain+0x2b2 10 fffff880022e04f0 fffff80001ed3f98 tcpip!FlReceiveNetBufferListChainCalloutRoutine+0xda 11 fffff880022e0540 fffff88001a53fe2 nt!KeExpandKernelStackAndCalloutEx+0xd8 12 fffff880022e0620 fffff88000fb20eb tcpip!FlReceiveNetBufferListChain+0xb2 13 fffff880022e0690 fffff88000f7bad6 NDIS!ndisMIndicateNetBufferListsToOpen+0xdb 14 fffff880022e0700 fffff88000efe5d4 NDIS!ndisMDispatchReceiveNetBufferLists+0x1d6 15 fffff880022e0b80 fffff88000efe549 NDIS!ndisMTopReceiveNetBufferLists+0x24 16 fffff880022e0bc0 fffff88000efe4e0 NDIS!ndisFilterIndicateReceiveNetBufferLists+0x29 17 fffff880022e0c00 fffff880039eeadb NDIS!NdisFIndicateReceiveNetBufferLists+0x50 18 fffff880022e0c40 fffff88000f170a7 nm3!NetmonReceiveNetBufferLists+0x25b \*\*\* ERROR: Symbol file could not be found. Defaulted to export symbols for vmxnet3n61x64.sys - 19 fffff880022e0d10 fffff88003e0c3a4 NDIS! ?? ::FNODOBFM::string’+0xcd8f
1a fffff880022e0d60 fffff88003e0d9f5 vmxnet3n61x64!realloc+0xabc0
1b fffff880022e0e30 fffff88000ef5951 vmxnet3n61x64!realloc+0xc211
1c fffff880022e0e70 fffff80001ed2f7c NDIS!ndisInterruptDpc+0x151
1d fffff880022e0f00 fffff80001ecb415 nt!KiRetireDpcList+0x1bc
1e fffff880022e0fb0 fffff80001ecb22c nt!KyRetireDpcList+0x5
1f fffff88009372be0 fffff80001f13253 nt!KiDispatchInterruptContinue
20 fffff88009372c10 fffff80001ec41c2 nt!KiDpcInterruptBypass+0x13
21 fffff88009372c20 000007fefd471205 nt!KiInterruptDispatch+0x212
22 000000000456f6d0 0000000000000000 0x000007fe`fd471205

8 fffff88002322180 fffffa80531b2470 ( 8) fffff8800232d1c0

8k

Child-SP RetAddr Call Site

00 fffff880092aed88 000007fed7ebc736 nt!KiSecondaryClockInterrupt
01 000000000204f390 0000000000000000 0x000007fe`d7ebc736

9 fffff88002393180 fffffa8035588060 ( 8) fffff8800239e1c0

9k

Child-SP RetAddr Call Site

00 fffff88008a99d88 000007ff000786fd nt!KiSecondaryClockInterrupt
01 000000000984edd8 0000000000000000 0x000007ff`000786fd

10 fffff880023c4180 fffffa804d863b50 ( 8) fffff880023cf1c0

10k

Child-SP RetAddr Call Site

00 fffff88008be93b0 fffff88002f11265 SRTSP64+0x91be3
01 fffff88008be9420 fffff88002f49607 SRTSP64+0x2f265
02 fffff88008be9490 fffff8800137f288 SRTSP64+0x67607
03 fffff88008be94e0 fffff8800137dd1b fltmgr!FltpPerformPostCallbacks+0x368
04 fffff88008be95b0 fffff8800139d2b9
fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x39b
05 fffff88008be9640 fffff800021c688b fltmgr!FltpCreate+0x2a9
06 fffff88008be96f0 fffff800021eb1df nt!IopParseDevice+0x14e2
07 fffff88008be9850 fffff800021c23ae nt!IopParseFile+0xaf
08 fffff88008be98b0 fffff800021c2e96 nt!ObpLookupObjectName+0x784
09 fffff88008be99b0 fffff800021c4c8c nt!ObOpenObjectByName+0x306
0a fffff88008be9a80 fffff800021d02b8 nt!IopCreateFile+0x2bc
0b fffff88008be9b20 fffff80001ec7553 nt!NtCreateFile+0x78
0c fffff88008be9bb0 000000007764c38a nt!KiSystemServiceCopyEnd+0x13
0d 00000000015fd0b8 0000000000000000 0x7764c38a

11 fffff88002481180 fffffa80351b4b50 ( 8) fffff8800248c1c0

11k

Child-SP RetAddr Call Site

00 fffff880024b0748 fffff880042a36a5 nt!KeBugCheckEx
01 fffff880024b0750 fffff80001ed34cc netft!NetftWatchdogTimerDpc+0xb9
02 fffff880024b07a0 fffff80001ed3366 nt!KiProcessTimerDpcTable+0x6c
03 fffff880024b0810 fffff80001ed324e nt!KiProcessExpiredTimerList+0xc6
04 fffff880024b0e60 fffff80001ed3037 nt!KiTimerExpiration+0x1be
05 fffff880024b0f00 fffff80001ecb415 nt!KiRetireDpcList+0x277
06 fffff880024b0fb0 fffff80001ecb22c nt!KyRetireDpcList+0x5
07 fffff8800a1af2c0 fffff80001f13253 nt!KiDispatchInterruptContinue
08 fffff8800a1af2f0 fffff80001ed36d8 nt!KiDpcInterruptBypass+0x13
09 fffff8800a1af300 fffff800021bd0b6 nt!KiSecondaryClockInterrupt+0x1a8
0a fffff8800a1af490 fffff800021bd57e nt!ObpWaitForMultipleObjects+0x13b
0b fffff8800a1af960 fffff80001ec7553 nt!NtWaitForMultipleObjects+0xe5
0c fffff8800a1afbb0 000000007764c3ea nt!KiSystemServiceCopyEnd+0x13
0d 00000000010ffb18 0000000000000000 0x7764c3ea

11: kd> !dpcs
CPU Type KDPC Function
7: Normal : 0xfffffa8025e0c118 0xfffff88000ef5800 NDIS!ndisInterruptDpc

11: Normal : 0xfffffa8024ed9118 0xfffff880012d0f78 ataport!IdePortCompletionDpc
11: kd> lmvm BHDrvx64
Browse full module list
start end module name
fffff88003c4a000 fffff88003de1000 BHDrvx64 (no symbols)
Loaded symbol image file: BHDrvx64.sys
Image path: ??\C:\ProgramData\Symantec\Symantec Endpoint
Protection\12.1.4100.4126.105\Data\Definitions\BASHDefs\20151015.011\BHDrvx64.sys
Image name: BHDrvx64.sys
Browse all global symbols functions data
Timestamp: Sat Sep 26 11:17:16 2015 (5605F21C)
CheckSum: 0019F93A
ImageSize: 00197000
File version: 9.3.0.69
Product version: 9.3.0.69
File flags: 8 (Mask 3F) Private
File OS: 40004 NT Win32
File type: 3.7 Driver
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Symantec Corporation
ProductName: BASH
InternalName: BashDriver
OriginalFilename: BHDrvx64.sys
ProductVersion: 9.3.0.69
FileVersion: 9.3.0.69
FileDescription: BASH Driver
LegalCopyright: Copyright (C) 2004 - 2015 Symantec Corporation. All
rights reserved.