How to debug a hang OS.

NTDEV
1

Hi

I hit a hang problem on my windows pv driver. We use microsoft ISCSI
initiator to setup ISCSI interface to access data. The whole process
will take about 10 hours, and the vm hang somewhere after the test
begain 5 hours. The hang state means no mouse and keyboard response. I
tried to use windbg to trace this problem, stop windbg when hanging, the
stack information is limited as following:

nt!DbgBreakPointWithStatus+0x4
nt!KeEnterCriticalRegion+0x8d
nt!KeQueryTickCount+0x391
nt!CcFlushCache+0x21a
nt!KeQueryTickCount+0x13d
nt!IoGetBaseFileSystemDeviceObject+0x76
nt!KeResetEvent+0xab7
nt!NtConnectPort+0x557
nt!KeQueryInterruptTime+0x1f2

I tried to use .crash to crash the vm, but no dump file generated. How
to locate this problem using windbg?
Any help is appreciated.

Thanks
Annie.

2

What did “!locks” tell you?

In a situation like this, you might have to actually look at every
thread on the system (“!process 0 7”). Yes, it generates a HUGE amount
of data, but that’s what it takes - poke around, look at the system, try
to figure out why the threads on it are not making forward progress.

The information you listed as a “stack trace” is unlike any stack trace
I’ve ever seen - I don’t see an EBP value, a return address, the
“parameters” list. Truthfully, it looks like you took a random
collection of interesting function names and shuffled them. Did you
REALLY get this as the output of the “kv” command?

Tony
OSR

3

.crash isn’t going to work unless your driver properly registered itself as
a crashdump handler, which is tricky when you’re pretending to be iSCSI.

As for debugging your hang, I suggest reading Windows Internals by Solomon
and Russinovich cover to cover. Once you’ve worked through the examples in
that book, you’ll have a good understanding of Windows debugging. (This is
actually the advice that I give to anyone new to Windows kernel
programming.)

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

“ANNIE LI” wrote in message news:xxxxx@ntdev…
> Hi
>
> I hit a hang problem on my windows pv driver. We use microsoft ISCSI
> initiator to setup ISCSI interface to access data. The whole process will
> take about 10 hours, and the vm hang somewhere after the test begain 5
> hours. The hang state means no mouse and keyboard response. I tried to use
> windbg to trace this problem, stop windbg when hanging, the stack
> information is limited as following:
>
> nt!DbgBreakPointWithStatus+0x4
> nt!KeEnterCriticalRegion+0x8d
> nt!KeQueryTickCount+0x391
> nt!CcFlushCache+0x21a
> nt!KeQueryTickCount+0x13d
> nt!IoGetBaseFileSystemDeviceObject+0x76
> nt!KeResetEvent+0xab7
> nt!NtConnectPort+0x557
> nt!KeQueryInterruptTime+0x1f2
>
> I tried to use .crash to crash the vm, but no dump file generated. How to
> locate this problem using windbg?
> Any help is appreciated.
>
> Thanks
> Annie.
>

4

Try to run “!stacks 2” This will show you all threads and their stack traces
and current state. Looking at the stack traces, you would be able to figure
out what has went wrong. If you cant figure out then you can also send the
out put on this thread.

Thanks,
Charan

On Wed, Jun 3, 2009 at 9:49 AM, ANNIE LI wrote:

> Hi
>
> I hit a hang problem on my windows pv driver. We use microsoft ISCSI
> initiator to setup ISCSI interface to access data. The whole process will
> take about 10 hours, and the vm hang somewhere after the test begain 5
> hours. The hang state means no mouse and keyboard response. I tried to use
> windbg to trace this problem, stop windbg when hanging, the stack
> information is limited as following:
>
> nt!DbgBreakPointWithStatus+0x4
> nt!KeEnterCriticalRegion+0x8d
> nt!KeQueryTickCount+0x391
> nt!CcFlushCache+0x21a
> nt!KeQueryTickCount+0x13d
> nt!IoGetBaseFileSystemDeviceObject+0x76
> nt!KeResetEvent+0xab7
> nt!NtConnectPort+0x557
> nt!KeQueryInterruptTime+0x1f2
>
> I tried to use .crash to crash the vm, but no dump file generated. How to
> locate this problem using windbg?
> Any help is appreciated.
>
> Thanks
> Annie.
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

5

Thanks.

What did “!locks” tell you?

I have set the symbol path in windbg like:
SRV*c:\symbol*http://msdl.microsoft.com/download/symbols.

But !locks shows

0: kd> !locks
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: nt!_ERESOURCE ***
*** ***
*************************************************************************
**** DUMP OF ALL RESOURCE OBJECTS ****
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: nt!_LIST_ENTRY ***
*** ***
*************************************************************************
808b8710: Unable to get value of ExpSystemResourcesList

Is there any error about the symbol path setting?

In a situation like this, you might have to actually look at every
thread on the system (“!process 0 7”). Yes, it generates a HUGE amount
of data, but that’s what it takes - poke around, look at the system, try
to figure out why the threads on it are not making forward progress.

The information you listed as a “stack trace” is unlike any stack trace
I’ve ever seen - I don’t see an EBP value, a return address, the
“parameters” list. Truthfully, it looks like you took a random
collection of interesting function names and shuffled them. Did you
REALLY get this as the output of the “kv” command?

Following is the stack info:

STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be
wrong.
f78beb5c 8084726c badb0d00 00000001 00000000 nt!DbgBreakPointWithStatus+0x4
f78bebdc 80844c83 00000006 028bebf0 c2371000 nt!KeEnterCriticalRegion+0x8d
f78bec80 80843eee c2372000 00006000 00000000 nt!KeQueryTickCount+0x391
f78becfc 80844a2f 00006000 00000000 00000001 nt!CcFlushCache+0x21a
f78bed40 8084328e 8a38adb0 808b70e0 8a3871f0 nt!KeQueryTickCount+0x13d
f78bed80 8082db10 8a3871f0 00000000 8a38adb0
nt!IoGetBaseFileSystemDeviceObject+0x76
f78bedac 80920833 8a3871f0 00000000 00000000 nt!KeResetEvent+0xab7
f78beddc 8083fe9f 8082da53 00000000 00000000 nt!NtConnectPort+0x557
00000000 00000000 00000000 00000000 00000000 nt!KeQueryInterruptTime+0x1f2

Thanks
Annie

6

You don’t seem to have the correct symbols for the os. Do the following commands and post the output:
!sym noisy
.reload /f nt
!lmi nt

What OS are you using? You can tell by:
vertarget

Calvin

— On Tue, 6/2/09, ANNIE LI wrote:

> From: ANNIE LI
> Subject: Re: [ntdev] How to debug a hang OS.
> To: “Windows System Software Devs Interest List”
> Date: Tuesday, June 2, 2009, 10:51 PM
> Thanks.
>
> > What did “!locks” tell you?? ?
> I have set the symbol path in windbg like: SRVc:\symbolhttp://msdl.microsoft.com/download/symbols.
>
> But !locks shows
>
> 0: kd> !locks
>
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? Your debugger is not using the correct
> symbols? ? ? ? ? ? ?
> ???
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? In order for this command to work
> properly, your symbol path???
>? ? must point to .pdb files that have full
> type information.? ? ?
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? Certain .pdb files (such as the public OS
> symbols) do not? ? ?
>? ? contain the required information.?
> Contact the group that? ? ?
>? ? provided you with these symbols if you
> need this command to? ?
>? ? work.? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? Type referenced: nt!_ERESOURCE?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ?
> ???
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>
> DUMP OF ALL RESOURCE OBJECTS
>
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? Your debugger is not using the correct
> symbols? ? ? ? ? ? ?
> ???
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? In order for this command to work
> properly, your symbol path???
>? ? must point to .pdb files that have full
> type information.? ? ?
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? Certain .pdb files (such as the public OS
> symbols) do not? ? ?
>? ? contain the required information.?
> Contact the group that? ? ?
>? ? provided you with these symbols if you
> need this command to? ?
>? ? work.? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>? ? Type referenced: nt!_LIST_ENTRY?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ?
>? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ???
>
> 808b8710: Unable to get value of ExpSystemResourcesList
>
> Is there any error about the symbol path setting?
> > In a situation like this, you might have to actually
> look at every
> > thread on the system (“!process 0
> 7”).???Yes, it generates a HUGE amount
> > of data, but that’s what it takes - poke around, look
> at the system, try
> > to figure out why the threads on it are not making
> forward progress.
> >
> > The information you listed as a “stack trace” is
> unlike any stack trace
> > I’ve ever seen - I don’t see an EBP value, a return
> address, the
> > “parameters” list.? Truthfully, it looks like you
> took a random
> > collection of interesting function names and shuffled
> them.? Did you
> > REALLY get this as the output of the “kv” command?
> >???
> Following is the stack info:
>
> STACK_TEXT: WARNING: Stack unwind information not
> available. Following frames may be wrong.
> f78beb5c 8084726c badb0d00 00000001 00000000
> nt!DbgBreakPointWithStatus+0x4
> f78bebdc 80844c83 00000006 028bebf0 c2371000
> nt!KeEnterCriticalRegion+0x8d
> f78bec80 80843eee c2372000 00006000 00000000
> nt!KeQueryTickCount+0x391
> f78becfc 80844a2f 00006000 00000000 00000001
> nt!CcFlushCache+0x21a
> f78bed40 8084328e 8a38adb0 808b70e0 8a3871f0
> nt!KeQueryTickCount+0x13d
> f78bed80 8082db10 8a3871f0 00000000 8a38adb0
> nt!IoGetBaseFileSystemDeviceObject+0x76
> f78bedac 80920833 8a3871f0 00000000 00000000
> nt!KeResetEvent+0xab7
> f78beddc 8083fe9f 8082da53 00000000 00000000
> nt!NtConnectPort+0x557
> 00000000 00000000 00000000 00000000 00000000
> nt!KeQueryInterruptTime+0x1f2
>
> Thanks
> Annie
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars
> visit: http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online
> at http://www.osronline.com/page.cfm?name=ListServer
>

7

Thanks.
The target machine is used by someone else now. I will post output once
i get it.

The version Win2k3 R2 3790 x86.

Thanks
Annie.

Calvin Guan wrote:

You don’t seem to have the correct symbols for the os. Do the following commands and post the output:
!sym noisy
.reload /f nt
!lmi nt

What OS are you using? You can tell by:
vertarget

Calvin

— On Tue, 6/2/09, ANNIE LI wrote:
>
>
>> From: ANNIE LI
>> Subject: Re: [ntdev] How to debug a hang OS.
>> To: “Windows System Software Devs Interest List”
>> Date: Tuesday, June 2, 2009, 10:51 PM
>> Thanks.
>>
>>
>>> What did “!locks” tell you?
>>>
>> I have set the symbol path in windbg like: SRVc:\symbolhttp://msdl.microsoft.com/download/symbols.
>>
>> But !locks shows
>>
>> 0: kd> !locks
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Your debugger is not using the correct
>> symbols
>>
>>
>>
>>
>>
>>
>> In order for this command to work
>> properly, your symbol path
>> must point to .pdb files that have full
>> type information.
>>
>>
>>
>>
>>
>> Certain .pdb files (such as the public OS
>> symbols) do not
>> contain the required information.
>> Contact the group that
>> provided you with these symbols if you
>> need this command to
>> work.
>>
>>
>>
>>
>>
>>
>>
>>
>> Type referenced: nt!_ERESOURCE
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> DUMP OF ALL RESOURCE OBJECTS
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Your debugger is not using the correct
>> symbols
>>
>>
>>
>>
>>
>>
>> In order for this command to work
>> properly, your symbol path
>> must point to .pdb files that have full
>> type information.
>>
>>
>>
>>
>>
>> Certain .pdb files (such as the public OS
>> symbols) do not
>> contain the required information.
>> Contact the group that
>> provided you with these symbols if you
>> need this command to
>> work.
>>
>>
>>
>>
>>
>>
>>
>>
>> Type referenced: nt!_LIST_ENTRY
>>
>>
>>
>>
>>
>>
>>
>>
>> 808b8710: Unable to get value of ExpSystemResourcesList
>>
>> Is there any error about the symbol path setting?
>>
>>> In a situation like this, you might have to actually
>>>
>> look at every
>>
>>> thread on the system (“!process 0
>>>
>> 7”). Yes, it generates a HUGE amount
>>
>>> of data, but that’s what it takes - poke around, look
>>>
>> at the system, try
>>
>>> to figure out why the threads on it are not making
>>>
>> forward progress.
>>
>>> The information you listed as a “stack trace” is
>>>
>> unlike any stack trace
>>
>>> I’ve ever seen - I don’t see an EBP value, a return
>>>
>> address, the
>>
>>> “parameters” list. Truthfully, it looks like you
>>>
>> took a random
>>
>>> collection of interesting function names and shuffled
>>>
>> them. Did you
>>
>>> REALLY get this as the output of the “kv” command?
>>>
>>>
>> Following is the stack info:
>>
>> STACK_TEXT: WARNING: Stack unwind information not
>> available. Following frames may be wrong.
>> f78beb5c 8084726c badb0d00 00000001 00000000
>> nt!DbgBreakPointWithStatus+0x4
>> f78bebdc 80844c83 00000006 028bebf0 c2371000
>> nt!KeEnterCriticalRegion+0x8d
>> f78bec80 80843eee c2372000 00006000 00000000
>> nt!KeQueryTickCount+0x391
>> f78becfc 80844a2f 00006000 00000000 00000001
>> nt!CcFlushCache+0x21a
>> f78bed40 8084328e 8a38adb0 808b70e0 8a3871f0
>> nt!KeQueryTickCount+0x13d
>> f78bed80 8082db10 8a3871f0 00000000 8a38adb0
>> nt!IoGetBaseFileSystemDeviceObject+0x76
>> f78bedac 80920833 8a3871f0 00000000 00000000
>> nt!KeResetEvent+0xab7
>> f78beddc 8083fe9f 8082da53 00000000 00000000
>> nt!NtConnectPort+0x557
>> 00000000 00000000 00000000 00000000 00000000
>> nt!KeQueryInterruptTime+0x1f2
>>
>> Thanks
>> Annie
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars
>> visit: http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online
>> at http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

8

Oh joy, those are so much fun to find.

The grunt method of debug:

Start with a known point, such as the IO processing entry, set a breakpoint, and use step, step into, or step around until you get to a function that hangs. With a little luck, and lots or restarts, you will eventually get to the point where you will find what is actually hanging. That may not necessarily answer WHY it’s hanging.

The grock method of debug:

Odds are good that it’s a spinlock problem. Check to see if there are multiple paths that acquire the spinlock and that all of those paths release the spinlock. Have you run pre-fast? CUV? Turn on Driver Verifier, and WDF Verifier? Use the !wdfxxxx commands to chase back WDF objects.

An aside: The !wdfxxx debug commands were some of the most useful functions I have used. In a matter of minutes I could find tons of information in my driver, from a cold break in. I used them heavily to find cancel IO problems with multiple threads feeding the IO queue. All I needed to know was the name of the driver.

Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of ANNIE LI
Sent: Tuesday, June 02, 2009 11:19 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] How to debug a hang OS.

Hi

I hit a hang problem on my windows pv driver. We use microsoft ISCSI
initiator to setup ISCSI interface to access data. The whole process
will take about 10 hours, and the vm hang somewhere after the test
begain 5 hours. The hang state means no mouse and keyboard response. I
tried to use windbg to trace this problem, stop windbg when hanging, the
stack information is limited as following:

nt!DbgBreakPointWithStatus+0x4
nt!KeEnterCriticalRegion+0x8d
nt!KeQueryTickCount+0x391
nt!CcFlushCache+0x21a
nt!KeQueryTickCount+0x13d
nt!IoGetBaseFileSystemDeviceObject+0x76
nt!KeResetEvent+0xab7
nt!NtConnectPort+0x557
nt!KeQueryInterruptTime+0x1f2

I tried to use .crash to crash the vm, but no dump file generated. How
to locate this problem using windbg?
Any help is appreciated.

Thanks
Annie.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

9

Hi

I used “!stacks 2” to show all stack information.
All status are blocked except for the following two:

c8.000108 899e7440 00007f4 RUNNING nt!RtlpBreakWithStatusInstruction
nt!KeUpdateSystemTime+0x12c
nt!KeFlushSingleTb+0x75
nt!MmSetAddressRangeModified+0x111

Ntfs!NtfsCreateNonresidentWithValue+0x272
Ntfs!NtfsConvertToNonresident+0x2ec
Ntfs!NtfsCommonWrite+0x105f
Ntfs!NtfsFsdWrite+0x16a
nt!IofCallDriver+0x45
fltMgr!FltpDispatch+0x152
nt!IofCallDriver+0x45
SYMEVENT+0x77a1
SYMEVENT+0x10e98
SYMEVENT+0x790b
SYMEVENT+0x7abf
nt!IofCallDriver+0x45
fltMgr!FltpDispatch+0x152
nt!IofCallDriver+0x45
nt!IopSynchronousServiceTail+0x10b
nt!NtWriteFile+0x663
nt!KiFastCallEntry+0xfc
ntdll!KiFastSystemCallRet
ntdll!NtWriteFile+0xc
kernel32!WriteFile+0xf7
WMIADAP!CGenerate::WriteToFile+0x6a
WMIADAP!CGenerate::ContentWrite+0x41

WMIADAP!CGenerate::GenerateFile_h+0x523

WMIADAP!WmiRefresherStuff::GenerateInternal+0xe1

WMIADAP!WmiRefresherStuff::Generate+0xb9

WMIADAP!DoReverseAdapterMaintenanceInternal+0xb5

WMIADAP!DoReverseAdapterMaintenance+0xf
WMIADAP!WinMain+0x472
WMIADAP!WinMainCRTStartup+0x182
kernel32!BaseProcessStart+0x23

[899d2778 activConsole.ex]
158.00016c 899debd0 0000815 RUNNING intelppm!AcpiC1Idle+0x12
+0xf772f000
nt!KiIdleLoop+0xa3

Does that mean a Ntfs error?

Charansing Rajput wrote:

Try to run “!stacks 2” This will show you all threads and their stack
traces and current state. Looking at the stack traces, you would be
able to figure out what has went wrong. If you cant figure out then
you can also send the out put on this thread.

Thanks,
Charan

On Wed, Jun 3, 2009 at 9:49 AM, ANNIE LI <annie.li>> http:@oracle.com http:> wrote:
>
> Hi
>
> I hit a hang problem on my windows pv driver. We use microsoft
> ISCSI initiator to setup ISCSI interface to access data. The whole
> process will take about 10 hours, and the vm hang somewhere after
> the test begain 5 hours. The hang state means no mouse and
> keyboard response. I tried to use windbg to trace this problem,
> stop windbg when hanging, the stack information is limited as
> following:
>
> nt!DbgBreakPointWithStatus+0x4
> nt!KeEnterCriticalRegion+0x8d
> nt!KeQueryTickCount+0x391
> nt!CcFlushCache+0x21a
> nt!KeQueryTickCount+0x13d
> nt!IoGetBaseFileSystemDeviceObject+0x76
> nt!KeResetEvent+0xab7
> nt!NtConnectPort+0x557
> nt!KeQueryInterruptTime+0x1f2
>
> I tried to use .crash to crash the vm, but no dump file generated.
> How to locate this problem using windbg?
> Any help is appreciated.
>
> Thanks
> Annie.
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging
> and other seminars visit: http://www.osr.com/seminars To unsubscribe,
> visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer</http:></http:></annie.li>

10

The full information of stack is too large, it is returned back. So here
is the link to get it.
http://os-drive.com/wiki/TestPage?action=AttachFile&do=get&target=stack

Thanks
Annie.

ANNIE LI wrote:

Hi

I used “!stacks 2” to show all stack information.
All status are blocked except for the following two:

c8.000108 899e7440 00007f4 RUNNING nt!RtlpBreakWithStatusInstruction
nt!KeUpdateSystemTime+0x12c
nt!KeFlushSingleTb+0x75
nt!MmSetAddressRangeModified+0x111

Ntfs!NtfsCreateNonresidentWithValue+0x272

Ntfs!NtfsConvertToNonresident+0x2ec
Ntfs!NtfsCommonWrite+0x105f
Ntfs!NtfsFsdWrite+0x16a
nt!IofCallDriver+0x45
fltMgr!FltpDispatch+0x152
nt!IofCallDriver+0x45
SYMEVENT+0x77a1
SYMEVENT+0x10e98
SYMEVENT+0x790b
SYMEVENT+0x7abf
nt!IofCallDriver+0x45
fltMgr!FltpDispatch+0x152
nt!IofCallDriver+0x45
nt!IopSynchronousServiceTail+0x10b
nt!NtWriteFile+0x663
nt!KiFastCallEntry+0xfc
ntdll!KiFastSystemCallRet
ntdll!NtWriteFile+0xc
kernel32!WriteFile+0xf7

WMIADAP!CGenerate::WriteToFile+0x6a

WMIADAP!CGenerate::ContentWrite+0x41

WMIADAP!CGenerate::GenerateFile_h+0x523

WMIADAP!WmiRefresherStuff::GenerateInternal+0xe1

WMIADAP!WmiRefresherStuff::Generate+0xb9

WMIADAP!DoReverseAdapterMaintenanceInternal+0xb5

WMIADAP!DoReverseAdapterMaintenance+0xf
WMIADAP!WinMain+0x472
WMIADAP!WinMainCRTStartup+0x182
kernel32!BaseProcessStart+0x23

[899d2778 activConsole.ex]
158.00016c 899debd0 0000815 RUNNING intelppm!AcpiC1Idle+0x12
+0xf772f000
nt!KiIdleLoop+0xa3

Does that mean a Ntfs error?

Charansing Rajput wrote:
> Try to run “!stacks 2” This will show you all threads and their stack
> traces and current state. Looking at the stack traces, you would be
> able to figure out what has went wrong. If you cant figure out then
> you can also send the out put on this thread.
>
> Thanks,
> Charan
>
>
> On Wed, Jun 3, 2009 at 9:49 AM, ANNIE LI <annie.li>>> http:@oracle.com http:> wrote:
>>
>> Hi
>>
>> I hit a hang problem on my windows pv driver. We use microsoft
>> ISCSI initiator to setup ISCSI interface to access data. The
>> whole process will take about 10 hours, and the vm hang somewhere
>> after the test begain 5 hours. The hang state means no mouse and
>> keyboard response. I tried to use windbg to trace this problem,
>> stop windbg when hanging, the stack information is limited as
>> following:
>>
>> nt!DbgBreakPointWithStatus+0x4
>> nt!KeEnterCriticalRegion+0x8d
>> nt!KeQueryTickCount+0x391
>> nt!CcFlushCache+0x21a
>> nt!KeQueryTickCount+0x13d
>> nt!IoGetBaseFileSystemDeviceObject+0x76
>> nt!KeResetEvent+0xab7
>> nt!NtConnectPort+0x557
>> nt!KeQueryInterruptTime+0x1f2
>>
>> I tried to use .crash to crash the vm, but no dump file
>> generated. How to locate this problem using windbg?
>> Any help is appreciated.
>>
>> Thanks
>> Annie.
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging
>> and other seminars visit: http://www.osr.com/seminars To unsubscribe,
>> visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer</http:></http:></annie.li>

11

More information about !process 0 7

http://os-drive.com/wiki/TestPage?action=AttachFile&do=get&target=process.txt

Thanks
Annie.

ANNIE LI wrote:

The full information of stack is too large, it is returned back. So
here is the link to get it.
http://os-drive.com/wiki/TestPage?action=AttachFile&do=get&target=stack

Thanks
Annie.

ANNIE LI wrote:
> Hi
>
> I used “!stacks 2” to show all stack information.
> All status are blocked except for the following two:
>
> c8.000108 899e7440 00007f4 RUNNING nt!RtlpBreakWithStatusInstruction
> nt!KeUpdateSystemTime+0x12c
> nt!KeFlushSingleTb+0x75
>
> nt!MmSetAddressRangeModified+0x111
>
> Ntfs!NtfsCreateNonresidentWithValue+0x272
>
> Ntfs!NtfsConvertToNonresident+0x2ec
> Ntfs!NtfsCommonWrite+0x105f
> Ntfs!NtfsFsdWrite+0x16a
> nt!IofCallDriver+0x45
> fltMgr!FltpDispatch+0x152
> nt!IofCallDriver+0x45
> SYMEVENT+0x77a1
> SYMEVENT+0x10e98
> SYMEVENT+0x790b
> SYMEVENT+0x7abf
> nt!IofCallDriver+0x45
> fltMgr!FltpDispatch+0x152
> nt!IofCallDriver+0x45
>
> nt!IopSynchronousServiceTail+0x10b
> nt!NtWriteFile+0x663
> nt!KiFastCallEntry+0xfc
> ntdll!KiFastSystemCallRet
> ntdll!NtWriteFile+0xc
> kernel32!WriteFile+0xf7
>
> WMIADAP!CGenerate::WriteToFile+0x6a
>
> WMIADAP!CGenerate::ContentWrite+0x41
>
> WMIADAP!CGenerate::GenerateFile_h+0x523
>
> WMIADAP!WmiRefresherStuff::GenerateInternal+0xe1
>
> WMIADAP!WmiRefresherStuff::Generate+0xb9
>
> WMIADAP!DoReverseAdapterMaintenanceInternal+0xb5
>
> WMIADAP!DoReverseAdapterMaintenance+0xf
> WMIADAP!WinMain+0x472
> WMIADAP!WinMainCRTStartup+0x182
> kernel32!BaseProcessStart+0x23
>
> [899d2778 activConsole.ex]
> 158.00016c 899debd0 0000815 RUNNING intelppm!AcpiC1Idle+0x12
> +0xf772f000
> nt!KiIdleLoop+0xa3
>
> Does that mean a Ntfs error?
>
>
>
> Charansing Rajput wrote:
>> Try to run “!stacks 2” This will show you all threads and their
>> stack traces and current state. Looking at the stack traces, you
>> would be able to figure out what has went wrong. If you cant figure
>> out then you can also send the out put on this thread.
>>
>> Thanks,
>> Charan
>>
>>
>> On Wed, Jun 3, 2009 at 9:49 AM, ANNIE LI <annie.li>>>> http:@oracle.com http:> wrote:
>>>
>>> Hi
>>>
>>> I hit a hang problem on my windows pv driver. We use microsoft
>>> ISCSI initiator to setup ISCSI interface to access data. The
>>> whole process will take about 10 hours, and the vm hang
>>> somewhere after the test begain 5 hours. The hang state means no
>>> mouse and keyboard response. I tried to use windbg to trace this
>>> problem, stop windbg when hanging, the stack information is
>>> limited as following:
>>>
>>> nt!DbgBreakPointWithStatus+0x4
>>> nt!KeEnterCriticalRegion+0x8d
>>> nt!KeQueryTickCount+0x391
>>> nt!CcFlushCache+0x21a
>>> nt!KeQueryTickCount+0x13d
>>> nt!IoGetBaseFileSystemDeviceObject+0x76
>>> nt!KeResetEvent+0xab7
>>> nt!NtConnectPort+0x557
>>> nt!KeQueryInterruptTime+0x1f2
>>>
>>> I tried to use .crash to crash the vm, but no dump file
>>> generated. How to locate this problem using windbg?
>>> Any help is appreciated.
>>>
>>> Thanks
>>> Annie.
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars
>>> visit: http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>>
>>>
>>> — NTDEV is sponsored by OSR For our schedule of WDF, WDM,
>>> debugging and other seminars visit: http://www.osr.com/seminars To
>>> unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer</http:></http:></annie.li>

12

Hi

I found that KernelTime, Working Set Sizes and PageFaultCount of process
Rtvscan.exe is larger than other process.

UserTime 00:00:03.640
KernelTime 00:00:10.843
QuotaPoolUsage[PagedPool] 207132
QuotaPoolUsage[NonPagedPool] 16120
Working Set Sizes (now,min,max) (19950, 50, 345) (79800KB, 200KB,
1380KB)
PeakWorkingSetSize 20099
PageFaultCount 290132

Furthermore, after i removed Symantec AntiVirus, this hang issue
disappear. The Symantec AntiVirus version is 10.1.7.7000 and the VM
version is Win2K3 SP2.
Anyone hit the similar problem? Are there any issue about this in
Symantec AntiVirus 10.1.7.7000?

Thanks
Annie.

ANNIE LI wrote:

More information about !process 0 7

http://os-drive.com/wiki/TestPage?action=AttachFile&do=get&target=process.txt

Thanks
Annie.

ANNIE LI wrote:
> The full information of stack is too large, it is returned back. So
> here is the link to get it.
> http://os-drive.com/wiki/TestPage?action=AttachFile&do=get&target=stack
>
> Thanks
> Annie.
>
> ANNIE LI wrote:
>> Hi
>>
>> I used “!stacks 2” to show all stack information.
>> All status are blocked except for the following two:
>>
>> c8.000108 899e7440 00007f4 RUNNING nt!RtlpBreakWithStatusInstruction
>> nt!KeUpdateSystemTime+0x12c
>> nt!KeFlushSingleTb+0x75
>>
>> nt!MmSetAddressRangeModified+0x111
>>
>> Ntfs!NtfsCreateNonresidentWithValue+0x272
>>
>> Ntfs!NtfsConvertToNonresident+0x2ec
>> Ntfs!NtfsCommonWrite+0x105f
>> Ntfs!NtfsFsdWrite+0x16a
>> nt!IofCallDriver+0x45
>> fltMgr!FltpDispatch+0x152
>> nt!IofCallDriver+0x45
>> SYMEVENT+0x77a1
>> SYMEVENT+0x10e98
>> SYMEVENT+0x790b
>> SYMEVENT+0x7abf
>> nt!IofCallDriver+0x45
>> fltMgr!FltpDispatch+0x152
>> nt!IofCallDriver+0x45
>>
>> nt!IopSynchronousServiceTail+0x10b
>> nt!NtWriteFile+0x663
>> nt!KiFastCallEntry+0xfc
>> ntdll!KiFastSystemCallRet
>> ntdll!NtWriteFile+0xc
>> kernel32!WriteFile+0xf7
>>
>> WMIADAP!CGenerate::WriteToFile+0x6a
>>
>> WMIADAP!CGenerate::ContentWrite+0x41
>>
>> WMIADAP!CGenerate::GenerateFile_h+0x523
>>
>> WMIADAP!WmiRefresherStuff::GenerateInternal+0xe1
>>
>> WMIADAP!WmiRefresherStuff::Generate+0xb9
>>
>> WMIADAP!DoReverseAdapterMaintenanceInternal+0xb5
>>
>> WMIADAP!DoReverseAdapterMaintenance+0xf
>> WMIADAP!WinMain+0x472
>> WMIADAP!WinMainCRTStartup+0x182
>> kernel32!BaseProcessStart+0x23
>>
>> [899d2778 activConsole.ex]
>> 158.00016c 899debd0 0000815 RUNNING intelppm!AcpiC1Idle+0x12
>> +0xf772f000
>> nt!KiIdleLoop+0xa3
>>
>> Does that mean a Ntfs error?
>>
>>
>>
>> Charansing Rajput wrote:
>>> Try to run “!stacks 2” This will show you all threads and their
>>> stack traces and current state. Looking at the stack traces, you
>>> would be able to figure out what has went wrong. If you cant figure
>>> out then you can also send the out put on this thread.
>>>
>>> Thanks,
>>> Charan
>>>
>>>
>>> On Wed, Jun 3, 2009 at 9:49 AM, ANNIE LI <annie.li>>>>> http:@oracle.com http:> wrote:
>>>>
>>>> Hi
>>>>
>>>> I hit a hang problem on my windows pv driver. We use microsoft
>>>> ISCSI initiator to setup ISCSI interface to access data. The
>>>> whole process will take about 10 hours, and the vm hang
>>>> somewhere after the test begain 5 hours. The hang state means
>>>> no mouse and keyboard response. I tried to use windbg to trace
>>>> this problem, stop windbg when hanging, the stack information
>>>> is limited as following:
>>>>
>>>> nt!DbgBreakPointWithStatus+0x4
>>>> nt!KeEnterCriticalRegion+0x8d
>>>> nt!KeQueryTickCount+0x391
>>>> nt!CcFlushCache+0x21a
>>>> nt!KeQueryTickCount+0x13d
>>>> nt!IoGetBaseFileSystemDeviceObject+0x76
>>>> nt!KeResetEvent+0xab7
>>>> nt!NtConnectPort+0x557
>>>> nt!KeQueryInterruptTime+0x1f2
>>>>
>>>> I tried to use .crash to crash the vm, but no dump file
>>>> generated. How to locate this problem using windbg?
>>>> Any help is appreciated.
>>>>
>>>> Thanks
>>>> Annie.
>>>>
>>>> —
>>>> NTDEV is sponsored by OSR
>>>>
>>>> For our schedule of WDF, WDM, debugging and other seminars
>>>> visit: http://www.osr.com/seminars
>>>>
>>>> To unsubscribe, visit the List Server section of OSR Online at
>>>> http://www.osronline.com/page.cfm?name=ListServer
>>>>
>>>>
>>>> — NTDEV is sponsored by OSR For our schedule of WDF, WDM,
>>>> debugging and other seminars visit: http://www.osr.com/seminars To
>>>> unsubscribe, visit the List Server section of OSR Online at
>>>> http://www.osronline.com/page.cfm?name=ListServer
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer</http:></http:></annie.li>

13

Not that specific issue, but I have set Driver Verifier on the
Symantic/Norton drivers. They failed. That plus the problems I did have with
Norton caused me to ban anything Symantec from my home computers.

The personal opinion of

Gary G. Little

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of ANNIE LI
Sent: Saturday, June 06, 2009 9:43 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] How to debug a hang OS.

Hi

I found that KernelTime, Working Set Sizes and PageFaultCount of process
Rtvscan.exe is larger than other process.

UserTime 00:00:03.640
KernelTime 00:00:10.843
QuotaPoolUsage[PagedPool] 207132
QuotaPoolUsage[NonPagedPool] 16120
Working Set Sizes (now,min,max) (19950, 50, 345) (79800KB, 200KB,
1380KB)
PeakWorkingSetSize 20099
PageFaultCount 290132

Furthermore, after i removed Symantec AntiVirus, this hang issue disappear.
The Symantec AntiVirus version is 10.1.7.7000 and the VM version is Win2K3
SP2.
Anyone hit the similar problem? Are there any issue about this in Symantec
AntiVirus 10.1.7.7000?

Thanks
Annie.

ANNIE LI wrote:

More information about !process 0 7

http://os-drive.com/wiki/TestPage?action=AttachFile
http:xt> &do=get&target=process.txt

Thanks
Annie.

ANNIE LI wrote:

The full information of stack is too large, it is returned back. So here is
the link to get it.
http://os-drive.com/wiki/TestPage?action=AttachFile
http:
&do=get&target=stack

Thanks
Annie.

ANNIE LI wrote:

Hi

I used “!stacks 2” to show all stack information.
All status are blocked except for the following two:

c8.000108 899e7440 00007f4 RUNNING nt!RtlpBreakWithStatusInstruction
nt!KeUpdateSystemTime+0x12c
nt!KeFlushSingleTb+0x75
nt!MmSetAddressRangeModified+0x111

Ntfs!NtfsCreateNonresidentWithValue+0x272
Ntfs!NtfsConvertToNonresident+0x2ec
Ntfs!NtfsCommonWrite+0x105f
Ntfs!NtfsFsdWrite+0x16a
nt!IofCallDriver+0x45
fltMgr!FltpDispatch+0x152
nt!IofCallDriver+0x45
SYMEVENT+0x77a1
SYMEVENT+0x10e98
SYMEVENT+0x790b
SYMEVENT+0x7abf
nt!IofCallDriver+0x45
fltMgr!FltpDispatch+0x152
nt!IofCallDriver+0x45
nt!IopSynchronousServiceTail+0x10b
nt!NtWriteFile+0x663
nt!KiFastCallEntry+0xfc
ntdll!KiFastSystemCallRet
ntdll!NtWriteFile+0xc
kernel32!WriteFile+0xf7
WMIADAP!CGenerate::WriteToFile+0x6a
WMIADAP!CGenerate::ContentWrite+0x41

WMIADAP!CGenerate::GenerateFile_h+0x523

WMIADAP!WmiRefresherStuff::GenerateInternal+0xe1

WMIADAP!WmiRefresherStuff::Generate+0xb9

WMIADAP!DoReverseAdapterMaintenanceInternal+0xb5

WMIADAP!DoReverseAdapterMaintenance+0xf
WMIADAP!WinMain+0x472
WMIADAP!WinMainCRTStartup+0x182
kernel32!BaseProcessStart+0x23

[899d2778 activConsole.ex]
158.00016c 899debd0 0000815 RUNNING intelppm!AcpiC1Idle+0x12
+0xf772f000
nt!KiIdleLoop+0xa3

Does that mean a Ntfs error?

Charansing Rajput wrote:

Try to run “!stacks 2” This will show you all threads and their stack traces
and current state. Looking at the stack traces, you would be able to figure
out what has went wrong. If you cant figure out then you can also send the
out put on this thread.

Thanks,
Charan

On Wed, Jun 3, 2009 at 9:49 AM, ANNIE LI wrote:

Hi

I hit a hang problem on my windows pv driver. We use microsoft ISCSI
initiator to setup ISCSI interface to access data. The whole process will
take about 10 hours, and the vm hang somewhere after the test begain 5
hours. The hang state means no mouse and keyboard response. I tried to use
windbg to trace this problem, stop windbg when hanging, the stack
information is limited as following:

nt!DbgBreakPointWithStatus+0x4
nt!KeEnterCriticalRegion+0x8d
nt!KeQueryTickCount+0x391
nt!CcFlushCache+0x21a
nt!KeQueryTickCount+0x13d
nt!IoGetBaseFileSystemDeviceObject+0x76
nt!KeResetEvent+0xab7
nt!NtConnectPort+0x557
nt!KeQueryInterruptTime+0x1f2

I tried to use .crash to crash the vm, but no dump file generated. How to
locate this problem using windbg?
Any help is appreciated.

Thanks
Annie.

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer</http:></http:>