Jen
if i run it from a bat file presetting _nt_symbol_path to exclude srv*
and url like below
the hang disappears
type *.bat
runchainedodbg.bat
set _nt_symbol_path=f:\symbols
ollydbg.exe ollydbg.exe msgbox.exe
if not for a few symbols i can at-least move forward
do you need anything to pinpoint it on symsrv.dll so that you can try
fixing it in some future drops
On 10/11/12, raj_r wrote:
> On 10/10/12, Jen-Lung Chiu wrote:
>> If the root cause if indeed wininet global mutex, this is not dbgeng
>> specific; instead all apps that use wininet would be affected. If other
>> apps
>> (that use wininet) work fine and only debuggers hang, then it could be
>> symsrv.dll specific.
>
>
> thanks for clarification it indeed must be symsrv specific as other
> networked apps run fine without any hang in fact when i posted
> yesterday and when i am posting this post now i have a hung instance
> side by side
>
> like below
>
> an output from hung local kernel debuging instance that cant get a stack
> trace
>
> lkd> !process 0 0 ollydbg.exe
> PROCESS 860415b0 SessionId: 0 Cid: 0e88 Peb: 7ffdd000 ParentCid: 00b4
> DirBase: 0fc801e0 ObjectTable: e3f2c858 HandleCount: 242.
> Image: ollydbg.exe
>
> PROCESS 85ea8a20 SessionId: 0 Cid: 0e98 Peb: 7ffdf000 ParentCid: 0e88
> DirBase: 0fc802e0 ObjectTable: e2f7a500 HandleCount: 121.
> Image: ollydbg.exe
>
> lkd> !process 0 0 msgbox.exe
> PROCESS 8649e500 SessionId: 0 Cid: 0f1c Peb: 7ffd4000 ParentCid: 0e98
> DirBase: 0fc80320 ObjectTable: e1dc1108 HandleCount: 5.
> Image: msgbox.exe
>
> lkd> .process /p /r 85ea8a20
> Implicit process is now 85ea8a20
> Loading User Symbols
> …
> lkd> !process 85ea8a20 17
> PROCESS 85ea8a20 SessionId: 0 Cid: 0e98 Peb: 7ffdf000 ParentCid: 0e88
> DirBase: 0fc802e0 ObjectTable: e2f7a500 HandleCount: 121.
> Image: ollydbg.exe
> VadRoot 86c96958 Vads 133 Clone 0 Private 1894. Modified 1152. Locked
> 0.
> DeviceMap e2f4b208
> Token e2e6acf8
> ElapsedTime 00:05:16.359
> UserTime 00:00:00.265
> KernelTime 00:00:00.796
> QuotaPoolUsage[PagedPool] 87820
> QuotaPoolUsage[NonPagedPool] 5640
> Working Set Sizes (now,min,max) (6720, 50, 345) (26880KB, 200KB,
> 1380KB)
> PeakWorkingSetSize 6751
> VirtualSize 60 Mb
> PeakVirtualSize 60 Mb
> PageFaultCount 11377
> MemoryPriority BACKGROUND
> BasePriority 8
> CommitCharge 4452
> DebugPort 864b1b10
>
> THREAD 85eacb18 Cid 0e98.0df4 Teb: 7ffde000 Win32Thread:
> e14b2720 WAITVerbose mode ON.
> Verbose mode OFF.
> Verbose mode ON.
> Verbose mode OFF.
> Verbose mode ON.
> Verbose mode OFF.
> Verbose mode ON.
>
>
> another lkd instance without symbolpath as suggested
>
> lkd> !process 85ea8a20 17
> PROCESS 85ea8a20 SessionId: 0 Cid: 0e98 Peb: 7ffdf000 ParentCid: 0e88
> DirBase: 0fc802e0 ObjectTable: e2f7a500 HandleCount: 121.
> Image: ollydbg.exe
> VadRoot 86c96958 Vads 133 Clone 0 Private 1894. Modified 1152. Locked
> 0.
> DeviceMap e2f4b208
> Token e2e6acf8
> ElapsedTime 02:00:01.187
> UserTime 00:00:00.265
> KernelTime 00:00:00.796
> QuotaPoolUsage[PagedPool] 87820
> QuotaPoolUsage[NonPagedPool] 5640
> Working Set Sizes (now,min,max) (6720, 50, 345) (26880KB, 200KB,
> 1380KB)
> PeakWorkingSetSize 6751
> VirtualSize 60 Mb
> PeakVirtualSize 60 Mb
> PageFaultCount 11377
> MemoryPriority BACKGROUND
> BasePriority 8
> CommitCharge 4452
> DebugPort 864b1b10
>
> THREAD 85eacb18 Cid 0e98.0df4 Teb: 7ffde000 Win32Thread:
> e14b2720 WAIT: (Suspended) KernelMode Non-Alertable
> FreezeCount 1
> 85eaccb4 Semaphore Limit 0x2
> Not impersonating
> DeviceMap e2f4b208
> Owning Process 0 Image:
> Attached Process 85ea8a20 Image: ollydbg.exe
> Wait Start TickCount 467348 Ticks: 449600
> (0:01:57:05.000)
> Context Switch Count 10742 LargeStack
> UserTime 00:00:00.250
> KernelTime 00:00:00.468
> Win32 Start Address ollydbg (0x00401000)
> Start Address kernel32!BaseProcessStartThunk (0x7c810705)
> Stack Init a8bdf000 Current a8bdec00 Base a8bdf000 Limit a8bda000
> Call 0
> Priority 9 BasePriority 8 PriorityDecrement 0 DecrementCount 16
> ChildEBP RetAddr Args to Child
> a8bdec18 80500cf0 85eacb88 85eacb18 804f9d72
> nt!KiSwapContext+0x2e (FPO: [Uses EBP] [0,0,4])
> a8bdec24 804f9d72 85eacc84 85eacb18 85eacb4c
> nt!KiSwapThread+0x46 (FPO: [0,0,0])
> a8bdec4c 80500ca2 00000000 00000005 00000000
> nt!KeWaitForSingleObject+0x1c2 (FPO: [Non-Fpo])
> a8bdec64 804fdb62 00000000 00000000 00000000
> nt!KiSuspendThread+0x18 (FPO: [3,0,0])
> a8bdecac 80500d0e 00000000 00000000 00000000
> nt!KiDeliverApc+0x124 (FPO: [Non-Fpo])
> a8bdecc4 804f9d72 00000000 00000000 00000000
> nt!KiSwapThread+0x64 (FPO: [0,0,0])
> a8bdecec 805b61a6 00000001 00000006 00131401
> nt!KeWaitForSingleObject+0x1c2 (FPO: [Non-Fpo])
> a8bded50 8053d658 000001ec 00000000 00000000
> nt!NtWaitForSingleObject+0x9a (FPO: [Non-Fpo])
> a8bded50 7c90e514 000001ec 00000000 00000000
> nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ a8bded64)
> 001314c4 7c90df5a 7c91b24b 000001ec 00000000
> ntdll!KiFastSystemCallRet (FPO: [0,0,0])
> 001314c8 7c91b24b 000001ec 00000000 00000000
> ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
> 00131550 7c901046 00e9f260 76e964dc 76e9f260
> ntdll!RtlpWaitForCriticalSection+0x132 (FPO: [Non-Fpo])
> 00131558 76e964dc 76e9f260 7c8099c0 00000000
> ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0])
> 0013158c 76e95ec2 00000000 00208e80 000009e4
> rasman!RemoteSubmitRequest+0x1c (FPO: [Non-Fpo])
> 001315a4 76e97220 00000000 00208e80 000009e4
> rasman!PutRequestInQueue+0x2b (FPO: [Non-Fpo])
> 001315ec 76e93dba 00000000 00000015 00131614
> rasman!SubmitRequest+0xd6 (FPO: [Non-Fpo])
> 00131624 76f0883b 00000000 00000000 00131648
> rasman!RasPortEnum+0x24b (FPO: [Non-Fpo])
> 0013164c 76f0db42 00000000 00131670 0013166c
> RASAPI32!GetRasPorts+0x50 (FPO: [Non-Fpo])
> 00131674 76f12489 00000000 001316dc 00000000
> RASAPI32!LoadPortsList2+0x39 (FPO: [Non-Fpo])
> 001316f4 76f13738 001319e4 00000008 00000000
> RASAPI32!ReadEntryList+0xdd5 (FPO: [Non-Fpo])
> 001319c4 76efdfbc 00131ca8 00000000 00000000
> RASAPI32!ReadPhonebookFile+0x2dd (FPO: [Non-Fpo])
> 00131a04 76ee4b8b 00131ca8 001f59c8 00131a44
> RASAPI32!DwEnumEntriesFromPhonebook+0x35 (FPO: [Non-Fpo])
> 00131eb8 76ee276a 00131efc 00000001 001f59c8
> RASAPI32!DwEnumEntriesInDir+0x1c9 (FPO: [Non-Fpo])
> 0013210c 76ee3d77 00000001 001f59c8 0013214c
> RASAPI32!DwEnumEntriesForPbkMode+0x9c (FPO: [Non-Fpo])
> 00132140 3d957634 00000000 00000000 001f59c8
> RASAPI32!RasEnumEntriesW+0xc2 (FPO: [Non-Fpo])
> 0013216c 3d957562 00000000 00000000 3d9e12fc
> WININET!RasEnumHelp::RasEnumHelp+0x55 (FPO: [Non-Fpo])
> 0013217c 3d9574e1 00000001 00000000 00000001
> WININET!DoConnectoidsExist+0x2a (FPO: [0,0,4])
> 001321a8 3d95749a 3d9e2fe4 3d9e12fc 00000001
> WININET!GetRasConnections+0x3d (FPO: [Non-Fpo])
> 001321c4 3d9498f4 00000000 001321f0 6298ef20
> WININET!IsDialUpConnection+0xb2 (FPO: [Non-Fpo])
> 00132210 3d93f85c 00000000 00000000 00000000
> WININET!FixProxySettingsForCurrentConnection+0x4a (FPO: [Non-Fpo])
> 00132e2c 3d966352 00000000 00000026 00000000
> WININET!InternetQueryOptionA+0xab3 (FPO: [Non-Fpo])
> 00133198 01fbe829 00000000 00000026 00000000
> WININET!InternetQueryOptionW+0x230 (FPO: [Non-Fpo])
> 001333cc 01fbe590 02006408 00000000 0000003c
> symsrv!StoreWinInet::dumpproxyinfo+0x40 (FPO: [Non-Fpo])
> 00133624 01fbe6fd 00000001 0013372c 01fb5afe
> symsrv!StoreWinInet::connect+0x1e5 (FPO: [Non-Fpo])
> 00133630 01fb5afe 00133eb8 00134fe8 00135408
> symsrv!StoreWinInet::find+0x23 (FPO: [Non-Fpo])
> 0013372c 01fb66f1 00000002 00133768 00133eb8
> symsrv!cascade+0x5a (FPO: [Non-Fpo])
> 00133e90 01fb64c5 00134524 00134fe8 00133eb8
> symsrv!SymbolServerByIndexW+0x17f (FPO: [4,463,4])
> 001340c8 0302c3fa 00134524 00134fe8 001349b0
> symsrv!SymbolServerW+0x73 (FPO: [Non-Fpo])
> 00134954 0301d4ab 001c3808 001351f8 00134dd8
> DBGHELP!symsrvGetFile+0x1e7 (FPO: [Non-Fpo])
> 00135628 0301e122 001e7e60 001e86f4 001e86e4
> DBGHELP!diaLocatePdb+0x5a8 (FPO: [Non-Fpo])
>
>
>
>
> i saw digest.dll mod load visible before hang in ollydbg and for sure
> this symbol less lkd instance shows it
>
>
>
>
> THREAD 864c0da8 Cid 0e98.0f48 Teb: 7ffdb000 Win32Thread:
> 00000000 WAIT: (Executive) KernelMode Non-Alertable
> a89a27d4 SynchronizationEvent
> Not impersonating
> DeviceMap e2f4b208
> Owning Process 0 Image:
> Attached Process 85ea8a20 Image: ollydbg.exe
> Wait Start TickCount 467348 Ticks: 449637
> (0:01:57:05.578)
> Context Switch Count 152
> UserTime 00:00:00.000
> KernelTime 00:00:00.000
> Win32 Start Address ntdll!RtlpWorkerThread (0x7c910250)
> Start Address kernel32!BaseThreadStartThunk (0x7c8106f9)
> Stack Init a89a3000 Current a89a2758 Base a89a3000 Limit a89a0000
> Call 0
> Priority 8 BasePriority 8 PriorityDecrement 0 DecrementCount 16
> ERROR: Symbol file could not be found. Defaulted to export
> symbols for C:\WINDOWS\system32\digest.dll -
> WARNING: Unable to verify checksum for C:\Program Files\Alwil
> Software\Avast5\snxhk.dll
> *** ERROR: Symbol file could not be found. Defaulted to export
> symbols for C:\Program Files\Alwil Software\Avast5\snxhk.dll -
> ChildEBP RetAddr Args to Child
> a89a2770 80500cf0 864c0e18 864c0da8 804f9d72
> nt!KiSwapContext+0x2e (FPO: [Uses EBP] [0,0,4])
> a89a277c 804f9d72 00000000 864c0da8 a89a27cc
> nt!KiSwapThread+0x46 (FPO: [0,0,0])
> a89a27a4 80638fc4 00000000 00000000 00000000
> nt!KeWaitForSingleObject+0x1c2 (FPO: [Non-Fpo])
> a89a2884 8063a099 85ea8a20 00000000 a89a28bc
> nt!DbgkpQueueMessage+0x17c (FPO: [Non-Fpo])
> a89a28a8 8063a1cb a89a28bc 00000001 a89a2d64
> nt!DbgkpSendApiMessage+0x45 (FPO: [Non-Fpo])
> a89a2934 804fcb42 a89a2d10 00000001 00000000
> nt!DbgkForwardException+0x8f (FPO: [Non-Fpo])
> a89a2cf4 8053e0a1 a89a2d10 00000000 a89a2d64
> nt!KiDispatchException+0x1f4 (FPO: [Non-Fpo])
> a89a2d5c 8053e7b1 0246f33c 75b07895 badb0d00
> nt!CommonDispatchException+0x4d (FPO: [0,20,0])
> a89a2d5c 75b07895 0246f33c 75b07895 badb0d00 nt!KiTrap03+0xad
> (FPO: [0,0] TrapFrame @ a89a2d64)
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> 0246f33c 7c91c4fa 75b07894 75b00000 00000001
> digest!EnumerateSecurityPackagesW+0x484b
>
>
>
>
>
>
>>
>> As a workaround, you could use debugger to attach to the hang app (with
>> null
>> .sympath setting), generate a dump with full memory (“.dump /mA
>> ”), process the dump from another machine. The needed PDBs
>> will
>> be downloaded from that session. From the live session you could set the
>> proper .sympath and continue.
>
>
> ok thanks let me check that out
>
>
>>
>> -----Original Message-----
>> From: xxxxx@lists.osr.com
>> [mailto:xxxxx@lists.osr.com] On Behalf Of raj_r
>> Sent: Tuesday, October 09, 2012 09:01 PM
>> To: Kernel Debugging Interest List
>> Subject: Re: RE:[windbg] windbg hangs when trying to view stack of
>> another
>> hung app on both attaching as well as attaching non invasive
>>
>> On 10/10/12, Jen-Lung Chiu wrote:
>>> Then this is expected. Most likely the hung app already holds the
>>> system-wide mutex in wininet.dll (ant not releases it yet), and this
>>> would hang all subsequent wininet requests (which symsrv.dll would
>>> call for http
>>> symsrv) because it could not grab that exact mutex.
>>
>>
>> thanks for replying back jen
>> is this dbgeng specific behavior ?
>>
>> i mean while there is a hang in the app i have many other network session
>> going on like ftp / web browser / wget etc and every one of them had
>> wininet
>> loaded they didnt hang ?
>>
>> is there an easy way to generate a list of files needed for the session
>> without having to kill a hang everytime and download the last visible pdb
>> ?
>>
>> i mean i downloaded 3 pdbs like you posted (msvcrt40.pdb ,
>> schannel.pdb,msapspXXX.pdb
>>
>> and killed windbg 3 times yesteday it went forward a bit and it is now
>> looking for someother pdb
>>
>> yep i know there is a manifest option in symchk
>>
>> but would it work for chained debuggers (yep i will try for sure and post
>> back i am just asking if you happen to have a ready made answer )
>>
>>
>>
>>
>>
>>>
>>> Workaround: try download the needed binaries and PDBs to local cache
>>> then restart the attach. As long as there is no need to hit the http
>>> symsrv store, it should be fine.
>>>
>>> -----Original Message-----
>>> From: xxxxx@lists.osr.com
>>> [mailto:xxxxx@lists.osr.com] On Behalf Of raj_r
>>> Sent: Tuesday, October 09, 2012 11:49 AM
>>> To: Kernel Debugging Interest List
>>> Subject: Re: RE:[windbg] windbg hangs when trying to view stack of
>>> another hung app on both attaching as well as attaching non invasive
>>>
>>> hung app is ollydbg an usermode debugger
>>>
>>> yes it loads wininet.dll
>>>
>>>
>>> 0:004> lm m wini*
>>> start end module name
>>> 3d930000 3da16000 WININET (deferred)
>>> 0:004> ~
>>> 0 Id: f18.e58 Suspend: 1 Teb: 7ffdf000 Unfrozen
>>> 1 Id: f18.dcc Suspend: 1 Teb: 7ffde000 Unfrozen
>>> 2 Id: f18.ed0 Suspend: 1 Teb: 7ffdc000 Unfrozen
>>> 3 Id: f18.c94 Suspend: 1 Teb: 7ffda000 Unfrozen . 4 Id: f18.d60
>>> Suspend: 1 Teb: 7ffdb000 Unfrozen 0:004> ~0kb ChildEBP RetAddr Args
>>> to Child
>>> 00132c10 7c90df5a 7c8025db 00000158 00000000 ntdll!KiFastSystemCallRet
>>> 00132c14 7c8025db 00000158 00000000 00000000
>>> ntdll!ZwWaitForSingleObject+0xc
>>> 00132c78 7c802542 00000158 ffffffff 00000000
>>> kernel32!WaitForSingleObjectEx+0xa8
>>> 00132c8c 3d9498e4 00000158 ffffffff bd0ec9bb
>>> kernel32!WaitForSingleObject+0x12
>>> 00132cd8 3d94f4af 00000000 00000000 001b16d0
>>> WININET!FixProxySettingsForCurrentConnection+0x3a
>>> 00132cf0 3d9457b3 00219808 00000000 00000000
>>> WININET!CFsm_HttpSendRequest::RunSM+0x61
>>> 00132d08 3d945c59 001b16d0 00000000 00000000 WININET!CFsm::Run+0x26
>>> 00132d20 3d974958 00219808 00000000 00000000 WININET!DoFsm+0x25
>>> 00132d48 3d94fafd 00000000 00000000 00000000
>>> WININET!HttpWrapSendRequest+0x136
>>> 00132d80 01faeb08 00cc000c 00000000 00000000
>>> WININET!HttpSendRequestW+0x5e
>>> 00132da0 01fae8db 12b71186 12b70b50 00000015
>>> symsrv!StoreWinInet::request+0xa8
>>> 00132dc8 01fae690 12b71186 12b70b50 00000001
>>> symsrv!StoreWinInet::fileinfo+0x19
>>> 00132ddc 01fab646 Verbose mode ON.
>>> Verbose mode OFF.
>>> Verbose mode ON.
>>> Verbose mode OFF.
>>> Verbose mode ON.
>>> Verbose mode OFF.
>>> Verbose mode ON.
>>> Verbose mode OFF.
>>>
>>>
>>> On 10/9/12, Jen-Lung Chiu wrote:
>>>> What is the “hung app”? Does it also load wininet.dll?
>>>>
>>>> -----Original Message-----
>>>> From: xxxxx@lists.osr.com
>>>> [mailto:xxxxx@lists.osr.com] On Behalf Of
>>>> xxxxx@gmail.com
>>>> Sent: Monday, October 08, 2012 10:46 PM
>>>> To: Kernel Debugging Interest List
>>>> Subject: RE:[windbg] windbg hangs when trying to view stack of
>>>> another hung app on both attaching as well as attaching non invasive
>>>>
>>>> does anyone have any ideas
>>>>
>>>> how to break this hang and go forward
>>>>
>>>> if i don’t let the debuggers to go to fetch symbols i don’t see the
>>>> hang
>>>>
>>>>
>>>>
>>>> —
>>>> WINDBG is sponsored by OSR
>>>>
>>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>>> http://www.osr.com/seminars
>>>>
>>>> To unsubscribe, visit the List Server section of OSR Online at
>>>> http://www.osronline.com/page.cfm?name=ListServer
>>>>
>>>>
>>>>
>>>> —
>>>> WINDBG is sponsored by OSR
>>>>
>>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>>> http://www.osr.com/seminars
>>>>
>>>> To unsubscribe, visit the List Server section of OSR Online at
>>>> http://www.osronline.com/page.cfm?name=ListServer
>>>>
>>>
>>> —
>>> WINDBG is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>>
>>>
>>>
>>> —
>>> WINDBG is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>>
>>
>> —
>> WINDBG is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>> —
>> WINDBG is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>