Hi Scott,
Here is more analysis from the input you asked and also from following
link:
http://www.osronline.com/showthread.cfm?link=173518
1: kd> !apc
*** Enumerating APCs in all processes
Process 8b781648 System
Process 8a465260 smss.exe
Process 8b6a3258 csrss.exe
Process 8ac94a88 winlogon.exe
Process 8af5cbf0 services.exe
Process 8af49258 lsass.exe
Process 8abadd88 svchost.exe
Process 8ac09d88 svchost.exe
Process 8ad67d88 svchost.exe
Process 8a544c10 svchost.exe
Process 8a547d88 svchost.exe
Process 8a4bfd88 spoolsv.exe
Process 8a516688 msdtc.exe
Process 8a4b6b18 aremote.exe
Process 8a482d88 cvpnd.exe
Process 8a4da5b8 dsm_sa_eventmgr
Process 8a4d6408 dsm_sa_datamgr3
Process 894b5d88 DicomServer.exe
Process 894ada08 DNTUS26.EXE
Process 894a88c8 DWRCS.EXE
Process 8b746c60 DxDmService.exe
Process 89471d88 svchost.exe
Process 89458688 inetinfo.exe
Process 8939ec60 mr2kserv.exe
Process 89399a08 dsm_om_shrsvc32
Process 8a49c020 extjob.exe
Process 89094720 TNSLSNR.EXE
Process 89054548 oracle.exe
Process 88f43308 svchost.exe
Process 88f50d88 dsm_om_connsvc3
Process 88f3b4a0 snmp.exe
Process 88f3a660 SynapseDBBackup
Process 88f2fd88 synapsedbstatis
Process 88f0a720 SynapsePreCache
Process 88f12818 SynapseUpdateMa
Process 88ed0020 lserver.exe
Process 88eec520 mqsvc.exe
Process 88e77720 svchost.exe
Process 88c46d18 wmiprvse.exe
Process 8a4ead88 svchost.exe
Process 88b8e4c8 svchost.exe
Process 88903780 wmiprvse.exe
Process 8884fd88 wmiprvse.exe
Process 88141ca8 DWRCST.EXE
Process 8813dd88 userinit.exe
Process 881219c8 userinit.exe
Process 8811cd88 explorer.exe
Thread 88124c00 ApcStateIndex 0 ApcListHead 88124c28 [KERNEL]
KAPC @ 88098a38
Type 12
KernelRoutine 8082159c nt!IopCompleteRequest+0
RundownRoutine 808f6782 nt!IopAbortRequest+0
1: kd> !thread 88124c00
THREAD 88124c00 Cid 03e4.0294 Teb: 7ffdd000 Win32Thread: e9e51348 WAIT:
(Unknown) KernelMode Non-Alertable
88bfef54 NotificationEvent
IRP List:
880989f8: (0006,01d8) Flags: 00000884 Mdl: 00000000
Not impersonating
DeviceMap e166b838
Owning Process 8811cd88 Image: explorer.exe
Attached Process N/A Image: N/A
Wait Start TickCount 6273 Ticks: 896 (0:00:00:14.000)
Context Switch Count 217 LargeStack
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address 0x010148a4
Start Address 0x77e617f8
Stack Init b9b2d6f0 Current b9b2c940 Base b9b2e000 Limit b9b28000 Call
b9b2d6f4
Priority 13 BasePriority 13 PriorityDecrement 0
ChildEBP RetAddr Args to Child
b9b2c958 80833491 88124c00 88124ca8 00000001 nt!KiSwapContext+0x26 (FPO:
[Uses EBP] [0,0,4])
b9b2c984 80829a82 88bfeef8 00000000 88506a70 nt!KiSwapThread+0x2e5 (FPO:
[Non-Fpo])
b9b2c9cc 808f9064 88bfef54 00000000 00000000 nt!KeWaitForSingleObject+0x346
(FPO: [Non-Fpo])
b9b2cac8 80937a20 8b1594b8 00000000 883507a8 nt!IopParseDevice+0xa4e (FPO:
[Non-Fpo])
b9b2cb48 80933b54 00000000 b9b2cb88 00000040 nt!ObpLookupObjectName+0x5b0
(FPO: [Non-Fpo])
b9b2cb9c 808eaeff 00000000 00000000 00000000 nt!ObOpenObjectByName+0xea
(FPO: [Non-Fpo])
b9b2cc18 808ec199 b9b2d13c 80100000 b9b2cfcc nt!IopCreateFile+0x447 (FPO:
[Non-Fpo])
b9b2cc74 808eec28 b9b2d13c 80100000 b9b2cfcc nt!IoCreateFile+0xa3 (FPO:
[Non-Fpo])
b9b2ccb4 808897cc b9b2d13c 80100000 b9b2cfcc nt!NtCreateFile+0x30 (FPO:
[Non-Fpo])
b9b2ccb4 8082e921 b9b2d13c 80100000 b9b2cfcc nt!KiFastCallEntry+0xfc (FPO:
[0,0] TrapFrame @ b9b2cce8)
b9b2cd58 ba7bedec b9b2d13c 80100000 b9b2cfcc nt!ZwCreateFile+0x11 (FPO:
[11,0,0])
WARNING: Stack unwind information not available. Following frames may be
wrong.
b9b2d0fc ba7a6d44 88131ef4 b9b2d13c 80100000 SYMTDI!rHeapFree+0x4484
b9b2d144 ba7abb71 88131ef4 00000000 e19c5be8
SYMTDI!ACMRegisterFilterModule+0x162c
b9b2d190 ba78d948 00000000 b9b2d1d0 ba78f6a6 SYMTDI!dereferenceModule+0x2d15
b9b2d19c ba78f6a6 b9b2d1d0 808ad204 e173abf0
SYMEVENT!EventObjectDestroy+0x338
b9b2d20c ba789f88 ea27c358 e19810d0 b9b2d250
SYMEVENT!EventObjectCreate+0x17d6
b9b2d230 80949da8 8a4306d0 000003e4 b9b2d2a0
SYMEVENT!SYMEvent_GetSubTask+0x318
e1667438 8b7e789e 00000000 e1674009 0c080603
nt!PsCallImageNotifyRoutines+0x36 (FPO: [Non-Fpo])
e1667448 61626453 003f005c 005c003f 003a0043 0x8b7e789e
e166744c 003f005c 005c003f 003a0043 0046005c 0x61626453
e1667450 005c003f 003a0043 0046005c 006a0075 0x3f005c
e1667454 003a0043 0046005c 006a0075 00560069 0x5c003f
e1667458 0046005c 006a0075 00560069 004e0050 0x3a0043
e166745c 006a0075 00560069 004e0050 0053005c 0x46005c
e1667460 00560069 004e0050 0053005c 0045004c 0x6a0075
e1667464 004e0050 0053005c 0045004c 00500045 0x560069
e1667468 0053005c 0045004c 00500045 0045002e 0x4e0050
e166746c 0045004c 00500045 0045002e 00450058 0x53005c
e1667470 00500045 0045002e 00450058 00000000 0x45004c
e1667474 0045002e 00450058 00000000 00000000 0x500045
e1667478 00450058 00000000 00000000 0c050608 0x45002e
e166747c 00000000 00000000 0c050608 7346744e 0x450058
So clearly explorer has APC queued in. The above stack does not look good
but using “dds b9b2c940 b9b2e000” does not gave any reasonable output
beyond
b9b2d190 ba78d948 00000000 b9b2d1d0 ba78f6a6 SYMTDI!dereferenceModule+0x2d15
so I was not able to get exact trace.
Dumping the IRP associated with this thread:
!irp 880989f8
Irp is active with 10 stacks 12 is current (= 00000000)
No Mdl: No System Buffer: Thread 88124c28: Irp is completed. Pending has
been returned
cmd flg cl Device File Completion-Context
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 8afb8718 00000000 ba789010-b9b2c858
\FileSystem\Ntfs SYMEVENT!SYMEvent_GetVMDataPtr
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 8a426020 00000000 b95325e0-88bcbb98
\Driver\SymEvent DxSpy!FilterCompletion
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 8947e8f8 00000000 00000000-00000000
\Driver\DxSpy
Args: 00000000 00000000 00000000 00000000
The thread 88124c28 listed in IRP output above indicates that this is not a
thread object so the thread is gone.
Also, dumping other IRP of blocked thread, I found following IRP Stack:
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[3,34] 0 e0 8b15d030 00000000 f721d558-8b15f0e8 Success Error Cancel
\Driver\Disk ftdisk!FtpRefCountCompletionRoutine
Args: 00001000 00000000 ff4eee00 00000035
[3, 0] 0 e1 8b15f030 00000000 f7b2aac2-b94ac0d0 Success Error Cancel
pending
\Driver\Ftdisk Ntfs!NtfsSingleSyncCompletionRoutine
Args: 00001000 00000000 ff4e7000 00000035
[3, 0] 0 0 8a435718 88ba99e0 00000000-00000000
\FileSystem\Ntfs
Args: 00001000 00000000 00004000 00000000
[3, 0] 0 0 8a4e63e8 88ba99e0 00000000-00000000
\Driver\SymEvent
Args: 00001000 00000000 00004000 00000000
So why many IRP’s are pending in \Driver\Disk?
Is there some sort of deadlock that explorer is not able to service APC?
Thanks for help
Ashish
On Mon, Dec 27, 2010 at 11:03 PM, Scott Noone wrote:
> That thread is just waiting for a non-cached I/O to complete from the
> storage branch. Are you sure this is the stuck thread? How long has it been
> waiting for? You can tell from the !thread output. Also, does !apc show any
> pending APCs for I/O completion?
>
> -scott
>
> –
> Scott Noone
> Consulting Associate
> OSR Open Systems Resources, Inc.
> http://www.osronline.com
>
> Hope to see you at the next OSR kernel debugging class February 14th in
> Columbia, MD!
>
> “Ashish Goyal” wrote in message
> news:xxxxx@ntfsd…
>
> Hi,
> We have a LEGACY filter driver (DxSpy) which archives files to secondary
> Storage. For that user marks a folder to be archived so all files in it are
> archived. Once of the user is seeing a problem of System Freezing when he
> tries to access one of the managed folder over Network. However, after some
> time system come back to normal. We have taken dump of system and here is
> the stack trace:
>
> b94b73e4 80833491 88f90db0 88f90e58 00000000 nt!KiSwapContext+0x26
> b94b7410 80829a82 878ace28 b94b7734 8b30c798 nt!KiSwapThread+0x2e5
> b94b7458 f7b2a9db b94b7670 00000000 00000000 nt!KeWaitForSingleObject+0x346
> b94b7478 f7b2a8e9 b94b7734 00001000 8b30c798 Ntfs!NtfsWaitSync+0x1c
> b94b7634 f7b2b156 b94b7734 878ace28 8b30c798 Ntfs!NtfsNonCachedIo+0x2f3
> b94b7720 f7b2b079 b94b7734 878ace28 00000001 Ntfs!NtfsCommonRead+0xaf5
> b94b78cc 8081df85 8a42d718 878ace28 8b504770 Ntfs!NtfsFsdRead+0x113
> b94b78e0 f7a2ec45 8b504770 8af5a008 88bfa030 nt!IofCallDriver+0x45
> b94b7908 8081df85 8ac32a88 878ace28 878ace28 fltmgr!FltpDispatch+0x6f
> b94b791c ba788f6b 00000000 8b4b0860 8081df85 nt!IofCallDriver+0x45
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> b94b793c b95347c1 00000000 88bfa030 000b0000
> SYMEVENT!SYMEvent_GetVMDataPtr+0x67cb
> b94b7970 b953190a 8b69a0d8 878ace28 0000094a DxSpy!CallAndRelease+0x91
> b94b79e0 8081df85 8b69a020 878ace28 878ace28 DxSpy!FilterDispatch+0x2fa
> b94b79f4 8081e50d 88f90db0 8b4c27f0 c0622780 nt!IofCallDriver+0x45
> b94b7a0c 80851176 8b37c70a 8b4c2828 8b4c2808 nt!IoPageRead+0x109
> b94b7aa8 8085ea9e 00000001 c44f0400 8b4c27f0 nt!MiDispatchFault+0xece
> b94b7b2c 8088c800 00000000 c44f0400 00000000 nt!MmAccessFault+0x89e
> b94b7b2c 808b64a6 00000000 c44f0400 00000000 nt!KiTrap0E+0xdc
> b94b7bf4 f7b6af2d 8b37c7b8 b94b7c24 00000400 nt!CcMapData+0x8c
> b94b7c14 f7b68494 87cd6ef8 8b30c798 000b0400 Ntfs!NtfsMapStream+0x4b
> b94b7c88 f7b6adf0 87cd6ef8 8a42d7f8 e3613ce0 Ntfs!NtfsReadMftRecord+0x86
> b94b7cc0 f7b6afac 87cd6ef8 8a42d7f8 e3613ce0 Ntfs!NtfsReadFileRecord+0x7a
> b94b7cf8 f7b5c88d 87cd6ef8 e3613cd8 e3613ce0
> Ntfs!NtfsLookupInFileRecord+0x37
> b94b7da8 f7b5e7be 87cd6ef8 00000001 e3613cd8
> Ntfs!NtfsUpdateFcbInfoFromDisk+0x3e
> b94b7e74 f7b5c0b9 87cd6ef8 8a62f3b0 8a62f51c Ntfs!NtfsOpenFile+0x330
> b94b8098 f7b6bef8 87cd6ef8 8a62f3b0 b94b80d8 Ntfs!NtfsCommonCreate+0x127e
> b94b819c 8081df85 8a42d718 8a62f3b0 8b504770 Ntfs!NtfsFsdCreate+0x17d
> b94b81b0 f7a3c458 00000000 8b504770 8abebc18 nt!IofCallDriver+0x45
> b94b81dc 8081df85 8ac32a88 8a62f3b0 b94b8240 fltmgr!FltpCreate+0xe4
> b94b81f0 ba7890c3 b94b8240 b9048200 00000000 nt!IofCallDriver+0x45
> b94b8270 808f904b b94b8418 8b5d0a00 00000000
> SYMEVENT!SYMEvent_GetVMDataPtr+0x6923
> b94b8358 80937a20 8b5d0a18 00000000 88354158 nt!IopParseDevice+0xa35
> b94b83d8 80933b54 00000000 b94b8418 00000040 nt!ObpLookupObjectName+0x5b0
> b94b842c 808eaeff 00000000 00000000 8b490000 nt!ObOpenObjectByName+0xea
> b94b84a8 808ec210 b94b85d8 00100008 b94b8538 nt!IopCreateFile+0x447
> b94b84f0 b952e20d b94b85d8 00100008 b94b8538
> nt!IoCreateFileSpecifyDeviceObjectHint+0x52
> b94b8574 b95303fc 00000000 8af66020 b94b85b0 DxSpy!OpenFile+0x1dd
> b94b8600 b9530901 00000000 8af66020 e3375000
> DxSpy!ReadMigrationInformation+0x15c
> b94b8644 b952920c 88925860 8af66020 e35ca000
> DxSpy!ReadMigrationInformationDir+0x161
> b94b870c b95324bd 8af66020 878b3e28 878b3fbb
> DxSpy!DirectoryCtrlCompleteCallback+0x65c
> b94b8758 b95326bb 88097d68 00000000 88097d68
> DxSpy!FilterCompletionWorkRoutine+0xbd
> b94b8780 8081e123 8b69a020 878b3e28 88097d68 DxSpy!FilterCompletion+0xdb
> b94b87b0 f7b2a1dc e3050c60 e1d0d0c0 b94b89d4 nt!IopfCompleteRequest+0xcd
> b94b87c0 f7b65e44 b94b8a24 878b3e28 00000000 Ntfs!NtfsCompleteRequest+0xc8
> b94b87d0 f7b664cb 878b3e28 878b3f94 88925860 Ntfs!NtfsQueryDirectory+0xcd5
> b94b89d4 f7b65daa b94b8a24 878b3e28 8a42d7f8 Ntfs!NtfsQueryDirectory+0xbc3
> b94b8a08 f7b65d15 b94b8a24 e35ac9f8 8ac32a88
> Ntfs!NtfsCommonDirectoryControl+0xbc
> b94b8b78 8081df85 8a42d718 878b3e28 8b504770
> Ntfs!NtfsFsdDirectoryControl+0xad
> b94b8b8c f7a2ec45 8b504770 8081cdde 8b32485c nt!IofCallDriver+0x45
> b94b8bb4 8081df85 8ac32a88 878b3e28 b94b8c10 fltmgr!FltpDispatch+0x6f
> b94b8bc8 ba788fd4 00000000 b94b8c10 8af5a008 nt!IofCallDriver+0x45
> b94b8c40 b95347c1 878b3fdc 878b4000 808ef9e0
> SYMEVENT!SYMEvent_GetVMDataPtr+0x6834
> b94b8c74 b95318d5 8b69a0d8 878b3e28 00000935 DxSpy!CallAndRelease+0x91
> b94b8ce4 8081df85 8b69a020 878b3e28 034eae7c DxSpy!FilterDispatch+0x2c5
> b94b8cf8 808f5511 b94b8d64 034eae7c 808ef9e0 nt!IofCallDriver+0x45
> b94b8d0c 808efa3d 8b69a020 878b3e28 88925860
> nt!IopSynchronousServiceTail+0x10b
> b94b8d30 808897cc 00000548 00000000 00000000 nt!NtQueryDirectoryFile+0x5d
> b94b8d30 7c82860c 00000548 00000000 00000000 nt!KiFastCallEntry+0xfc
> 034eaec4 00000000 00000000 00000000 00000000 0x7c82860c
>
>
> There are multiple threads with above stack trace. So each thread is
> waiting on different notification event
>
> dt _DISPATCHER_HEADER b94b7670
> nt!_DISPATCHER_HEADER
> +0x000 Type : 0 ‘’
> +0x001 Absolute : 0 ‘’
> +0x001 NpxIrql : 0 ‘’
> +0x002 Size : 0x4 ‘’
> +0x002 Hand : 0x4 ‘’
> +0x003 Inserted : 0 ‘’
> +0x003 DebugActive : 0 ‘’
> +0x000 Lock : 0n262144
> +0x004 SignalState : 0n0
> +0x008 WaitListHead : _LIST_ENTRY [0x88f90e58 - 0x88f90e58]
>
> Can anybody suggest from stack Trace the reason of system Freeze?
>
> Thanks for help
> Ashish
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>