Generating a manual crash dump when system is hung

Hi,

I am investigating an issue where the system gets hung when it is
shuting down (I can see the “shutting down” text on the screen). I
have attached windbg and from there tried to break but its not
possible. I tried to take a manual crash dump but that is not possible
either. Anyone has any ideas on how to force a manual crash dump?

Breaking and generating crash dumps under normal conditions is not a problem.

Thanks
Faik

There is the ps2 keyboard mechanism - that works if the system is responding
to interrupts and has a ps2 keyboard. Otherwise we have discussed the ‘nmi
generator’ pci device here - which requires one to actually put together
some hardware (or maybe OSR will sell you one, I forget.)

Mark Roddy

On Tue, Apr 27, 2010 at 11:18 AM, Faik Riza wrote:

> Hi,
>
> I am investigating an issue where the system gets hung when it is
> shuting down (I can see the “shutting down” text on the screen). I
> have attached windbg and from there tried to break but its not
> possible. I tried to take a manual crash dump but that is not possible
> either. Anyone has any ideas on how to force a manual crash dump?
>
> Breaking and generating crash dumps under normal conditions is not a
> problem.
>
> Thanks
> Faik
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

>or maybe OSR will sell you one, I forget

Used to sell them, but we don’t anymore.

-scott


Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Mark Roddy” wrote in message news:xxxxx@ntdev…
There is the ps2 keyboard mechanism - that works if the system is responding
to interrupts and has a ps2 keyboard. Otherwise we have discussed the ‘nmi
generator’ pci device here - which requires one to actually put together
some hardware (or maybe OSR will sell you one, I forget.)

Mark Roddy

On Tue, Apr 27, 2010 at 11:18 AM, Faik Riza wrote:

Hi,

I am investigating an issue where the system gets hung when it is
shuting down (I can see the “shutting down” text on the screen). I
have attached windbg and from there tried to break but its not
possible. I tried to take a manual crash dump but that is not possible
either. Anyone has any ideas on how to force a manual crash dump?

Breaking and generating crash dumps under normal conditions is not a
problem.

Thanks
Faik


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

I believe that USB keyboards will also work, though not as well as a PS/2 keyboard, due the stack running at a different IRQL. They also involve setting different registry keys, I think. If interested, Google is your friend here.

Either way, if I understand you correctly, I’m surprised that you can’t break in that soon in the shutdown process, but apparently, that’s the way it is.

If you get really desperate, in a worst case scenario (as in $$$), you could connect a JTAG emulator (like an Arium ECM-50 or ECM-XDP).

Have you tried installing a CHK build and enabling every nt!Kd_*_Mask under the sun? It very well might not help much - and if you can’t reproduce it, not at all - but barring your windbg issue getting fixed or procuring some hardware assistance, it would seem like your only avenue, at least for the moment.

Also, traces rock.

The other thing that I’ll throw out as a long shot is that if you’re using anything other than the serial kd transport, I’d give that a try.

Q:

  1. What’s your target os?

  2. Does it have a hypervisor running?

  3. What verison of windbg?

  4. What kd transport are you using?

mm

Is this on Windows 7? Do you have a driver installed? If you have a driver,
set breakpoints in your code, and make sure you have released all IRPs.
Triple check the IRP’s. An uncompleted IRP can cause this.

If it is Win7 and you don’t have your own driver running, it may be what I
see now and then with Win7, which is caused by an uncompleted IRP in one of
the network/NDIS drivers, wireless I believe. Just let it run for at least
an hour. Somewhere around 30 to 45 minutes it will BSOD with the uncompleted
IRP information, because eventually the IRP timer will trigger and the
system will then go down the panic path and present that lovely shade of
blue we all know and love.

Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Faik Riza
Sent: Tuesday, April 27, 2010 10:19 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Generating a manual crash dump when system is hung

Hi,

I am investigating an issue where the system gets hung when it is
shuting down (I can see the “shutting down” text on the screen). I
have attached windbg and from there tried to break but its not
possible. I tried to take a manual crash dump but that is not possible
either. Anyone has any ideas on how to force a manual crash dump?

Breaking and generating crash dumps under normal conditions is not a
problem.

Thanks
Faik


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

__________ Information from ESET Smart Security, version of virus signature
database 5065 (20100427) __________

The message was checked by ESET Smart Security.

http://www.eset.com

__________ Information from ESET Smart Security, version of virus signature
database 5065 (20100427) __________

The message was checked by ESET Smart Security.

http://www.eset.com

Hi,

Tried with ps2 keyboard (laptop keyboards are ps2 if I am not
mistaken), have tried with USB keyboard as well, but when using a USB
keyboard I just can’t reproduce the issue anymore.
Also the issue isn’t reproducible when enabling verifier. Something
timing related. I guess that having a checked build of the OS will
just result in me not being able to repro any more, hence why I
haven’t tried this yet.

Haven’t tried debugging using a serial cable, don’t have that
interface on the laptop. Can’t see how that would have helped though.

Forgot to mention OS versions and such in my original post.

OS version: Win 7
No hyper-visor.
Latest Windbg version, the one found with the WDK this days.
I am using 1394 as KD transport.

I have several drivers running in the system (some developed by me
some by others), problem with the breakpoint is that I have no idea
when the freeze is happening checking the IRP’s before this would be
of no use I guess.

Hmm I guess that my only option is to use what mobrien suggested.
Still hoping for some other way (easier) to force a crash on the
system though.

Thanks for taking time with this.

Cheers
Faik

Yeah, I don’t think that using the serial transport will likely help either, but it was windbg’s original (and up until really quite recently, only) transport and it definitely remains its most reliable. Also, while I can’t really say that it has been clearly demonstrated that the past couple version’s 1394 drivers have had some issues, I know several people that I trust (as well as myself) who feel that some look like they very well might exist, since they windbg team completely changed the 1394 driver, only about a year and one half ago, in the process breaking compatibility with the old one. If judged by the installation issues, it was not exactly a smooth process.

  1. One not especially appealing idea I have is that you could sort of shell sort you way through this problem by registering for shutdown notifications at various points (callbacks, et. c.) and break in those routines; it should at least get you closer to the origin.

  2. Also, you might try attaching windbg to winlogon and/or csrss.

  3. As a one time thing, you might try enabling kd line protocol diagnostics (ctrl+alt+d) and see if that can tell you anything about why you can’t break in.

  4. Another thing you could try that I’m not sure will help or not is to set an AMLI breakpoint using the amli debugger in windbg (this requires the CHK build of acpi.sys).

  5. Finally, since you’re running on Win 7 (really anything but Vista/S2K8), you could use .kdfiles to try out CHK builds of whatever you wanted (kernel, hal, et. c.) without having to install an OS or go through the bother of editing the registry to create a partial CHK build.

Good luck,

mm

Did you confirm that you configured the ps2 driver to crash? Ie can you initiate a crash from the keyboard when the machine is running normally?

d

sent from a phpne with no keynoard

-----Original Message-----
From: Faik Riza
Sent: April 27, 2010 1:19 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Generating a manual crash dump when system is hung

Hi,

Tried with ps2 keyboard (laptop keyboards are ps2 if I am not
mistaken), have tried with USB keyboard as well, but when using a USB
keyboard I just can’t reproduce the issue anymore.
Also the issue isn’t reproducible when enabling verifier. Something
timing related. I guess that having a checked build of the OS will
just result in me not being able to repro any more, hence why I
haven’t tried this yet.

Haven’t tried debugging using a serial cable, don’t have that
interface on the laptop. Can’t see how that would have helped though.

Forgot to mention OS versions and such in my original post.

OS version: Win 7
No hyper-visor.
Latest Windbg version, the one found with the WDK this days.
I am using 1394 as KD transport.

I have several drivers running in the system (some developed by me
some by others), problem with the breakpoint is that I have no idea
when the freeze is happening checking the IRP’s before this would be
of no use I guess.

Hmm I guess that my only option is to use what mobrien suggested.
Still hoping for some other way (easier) to force a crash on the
system though.

Thanks for taking time with this.

Cheers
Faik


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

@mm

  1. Yes this might be an approach.
  2. Will try this too.
  3. Don’t know what this is will have to look it up and try it as well.
  4. Thought about AMLI debugging before but for some reason I abandoned that.
  5. I guess I need to start looking at the checked buids of various components.

@Doron
Yes I confirmed that I can force the machine to crash when it is
running normally. Breaking in with WinDbg is working fine as well.

I probably should mention that my device is a USB device and that the
tests we are doing are restart tests.

Cheers
Faik