how do I generate crash dump in a hung machine

Hi,

I have a x86 Win7 machine, where I can repro a system hang (no mouse and
keyboard) using one of our own drivers. The machine does not have a COM or
IEEE 1394 port, so I can’t debug as the hang happens. When the machine
hangs, it does not generate the crash dump. I played with all crash dump
generation option under system settings, e.g., Setting “Complete Crash Dump”
etc. That didn’t help.

I also tried setting CrashOnCtrlScroll registry value under service
“parameters” for both i8042prt.sys and kbhid.sys, and hit Ctrl+Scroll+Scroll
during hang or during normal no-hang scenarios. Still no crash dump
generated.

Is there a reliable way I can generate the crash dump on the hung x86
Win7 machine?

Thanks,
Chandra

http://www.connecttech.com/sub/products/PCI_DumpSwitch.asp

Google PCI Dump Switch. You have to have a PCI slot. There is actually
no guarantee that this will work either, but it is not expensive and
you do not seem to have a lot of alternatives.

p.s. I though OSR was selling one of these for a while, I didn’t see it.

Mark Roddy

On Thu, Dec 30, 2010 at 9:21 AM, chandra97 97 wrote:
> Hi,
>
> I have a x86 Win7 machine, where I can repro a system hang (no mouse and
> keyboard) using one of?our own drivers. The machine does not have a COM or
> IEEE 1394 port, so I can’t debug as the hang happens. When the machine
> hangs, it does not generate the crash dump. I played with all crash dump
> generation option under system settings, e.g., Setting “Complete Crash Dump”
> etc. That didn’t help.
>
> I also tried setting CrashOnCtrlScroll registry value under service
> “parameters” for both i8042prt.sys and kbhid.sys, and hit Ctrl+Scroll+Scroll
> during hang or during normal no-hang scenarios. Still no crash dump
> generated.
>
> Is there a reliable way I can generate the crash dump on the hung x86
> Win7?machine?
>
> Thanks,
> Chandra
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
> List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

Use ctrl on the right side.
I hav forced crash dump using ctrl-scroll combo once.
Left side ctrl never worked for me.

Regards
Deepak

On Thu, Dec 30, 2010 at 7:51 PM, chandra97 97 wrote:

> Hi,
>
> I have a x86 Win7 machine, where I can repro a system hang (no mouse and
> keyboard) using one of our own drivers. The machine does not have a COM or
> IEEE 1394 port, so I can’t debug as the hang happens. When the machine
> hangs, it does not generate the crash dump. I played with all crash dump
> generation option under system settings, e.g., Setting “Complete Crash Dump”
> etc. That didn’t help.
>
> I also tried setting CrashOnCtrlScroll registry value under service
> “parameters” for both i8042prt.sys and kbhid.sys, and hit Ctrl+Scroll+Scroll
> during hang or during normal no-hang scenarios. Still no crash dump
> generated.
>
> Is there a reliable way I can generate the crash dump on the hung x86
> Win7 machine?
>
> Thanks,
> Chandra
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

We did sell them at one time, but discontinued the item due to both technical and customer satisfaction issues. It is difficult for any dump switch to work all the time and on all flavors of systems. Also, 5V PCI is rapidly disappearing from the scene.

Unless you know the switch will work for you, I suggest you find another method.

Peter
OSR

>Use ctrl on the right side.
OK. Using right side Ctrl works when machine is responsive. But, when the
machine is hung that doesn’t work.

Google PCI Dump Switch. You have to have a PCI slot. There is actually
no guarantee that this will work either,

Given you said it may not work and also don’t think its a good idea to ask
the vendor to add that switch for us, so we can debug!

Any other suggestions? Given all latest laptop models may not have COM or
firewire ports, its disappointing that OS doesn’t provide a convenient way
to generate dump for such hung machine cases. The hung case only happens for
a specific vendor laptop and does not happen with other test laptops we
have.

I’ll try USB 2.0 cable debugging next. The windbg documenation says lots of
contraints in using it. I’ll see how it goes

Thanks,
Chandra
On Thu, Dec 30, 2010 at 11:19 AM, Deepak Gupta wrote:

> Use ctrl on the right side.
> I hav forced crash dump using ctrl-scroll combo once.
> Left side ctrl never worked for me.
>
> Regards
> Deepak
>
> On Thu, Dec 30, 2010 at 7:51 PM, chandra97 97 wrote:
>
>> Hi,
>>
>> I have a x86 Win7 machine, where I can repro a system hang (no mouse and
>> keyboard) using one of our own drivers. The machine does not have a COM or
>> IEEE 1394 port, so I can’t debug as the hang happens. When the machine
>> hangs, it does not generate the crash dump. I played with all crash dump
>> generation option under system settings, e.g., Setting “Complete Crash Dump”
>> etc. That didn’t help.
>>
>> I also tried setting CrashOnCtrlScroll registry value under service
>> “parameters” for both i8042prt.sys and kbhid.sys, and hit Ctrl+Scroll+Scroll
>> during hang or during normal no-hang scenarios. Still no crash dump
>> generated.
>>
>> Is there a reliable way I can generate the crash dump on the hung x86
>> Win7 machine?
>>
>> Thanks,
>> Chandra
>>
>> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
>> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
>> the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

chandra97 97 wrote:

Hi,

I have a x86 Win7 machine, where I can repro a system hang (no mouse
and keyboard) using one of our own drivers. The machine does not have
a COM or IEEE 1394 port, so I can’t debug as the hang happens. When
the machine hangs, it does not generate the crash dump. I played with
all crash dump generation option under system settings, e.g., Setting
“Complete Crash Dump” etc. That didn’t help.

Of course not. Crash dumps occur when there is a bug check, which is a
more-or-less controlled shutdown. When your machine hangs, nothing can run.

Does your driver involve a piece of hardware? This kind of hang can
signal an interrupt storm, or a PCI bus protocol error. It can also be
as simple as a driver in an infinite loop at raised IRQL.

Do you have a free PCI slot? You can get PCI cards with 1394 ports. I
keep one on hand for machines like this.

I also tried setting CrashOnCtrlScroll registry value under service
“parameters” for both i8042prt.sys and kbhid.sys, and hit
Ctrl+Scroll+Scroll during hang or during normal no-hang scenarios.
Still no crash dump generated.

This can only work if interrupts are still running.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

chandra97 97 wrote:

Any other suggestions? Given all latest laptop models may not have COM
or firewire ports, its disappointing that OS doesn’t provide a
convenient way to generate dump for such hung machine cases. The hung
case only happens for a specific vendor laptop and does not happen
with other test laptops we have.

There has never been a way to do this reliably for hung machines. In
order to force a dump, you have to be able to get the processor’s
attention. That requires an interrupt. If interrupts are already being
blocked, then nothing on earth is going to break in.

Have you seen this on more than one instance of this model? You might
simply have a hardware problem – defective RAM, or a bad trace on the
motherboard.

I’ll try USB 2.0 cable debugging next. The windbg documenation says
lots of contraints in using it. I’ll see how it goes

If your USB keyboard can’t get interrupts, then the kernel debugger
interface can’t use USB, either. Making a dump requires that the system
be at least nominally operational.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

“chandra97 97” wrote in message news:xxxxx@ntdev:

>
> I’ll try USB 2.0 cable debugging next. The windbg documenation says lots of
> contraints in using it. I’ll see how it goes
>

If you haven’t already wasted your money on the cable, you may want to
look at the paper
http://www.microsoft.com/whdc/driver/foundation/drvdev_intro.mspx There
is a procedure for knowing in some cases the system will not work with
USB, unfortunately there is no way to ensure a system will work with USB
before trying it. Even if it does work it is less than reliable, and it
is doubtful it will help with a hang.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

It sounds like your last hope here is to enable writes to the Event Viewer, and start adding event log writes at key points until you have your “Eureka” moment, or you simply suggest the customer take a large sledge hammer to that laptop. You could always ask the customer to ship it to you and then bribe the carrier to “loose” it.

Gary G. Little

----- Original Message -----
From: “chandra97 97”
To: “Windows System Software Devs Interest List”
Sent: Thursday, December 30, 2010 11:42:53 AM
Subject: Re: [ntdev] how do I generate crash dump in a hung machine

>Use ctrl on the right side.
OK. Using right side Ctrl works when machine is responsive. But, when the machine is hung that doesn’t work.

>Google PCI Dump Switch. You have to have a PCI slot. There is actually
no guarantee that this will work either,

Given you said it may not work and also don’t think its a good idea to ask the vendor to add that switch for us, so we can debug!

Any other suggestions? Given all latest laptop models may not have COM or firewire ports, its disappointing that OS doesn’t provide a convenient way to generate dump for such hung machine cases. The hung case only happens for a specific vendor laptop and does not happen with other test laptops we have.

I’ll try USB 2.0 cable debugging next. The windbg documenation says lots of contraints in using it. I’ll see how it goes

Thanks,
Chandra

On Thu, Dec 30, 2010 at 11:19 AM, Deepak Gupta < xxxxx@gmail.com > wrote:

Use ctrl on the right side.
I hav forced crash dump using ctrl-scroll combo once.
Left side ctrl never worked for me.

Regards
Deepak

On Thu, Dec 30, 2010 at 7:51 PM, chandra97 97 < xxxxx@gmail.com > wrote:

Hi,

I have a x86 Win7 machine, where I can repro a system hang (no mouse and keyboard) using one of our own drivers. The machine does not have a COM or IEEE 1394 port, so I can’t debug as the hang happens. When the machine hangs, it does not generate the crash dump. I played with all crash dump generation option under system settings, e.g., Setting “Complete Crash Dump” etc. That didn’t help.

I also tried setting CrashOnCtrlScroll registry value under service “parameters” for both i8042prt.sys and kbhid.sys, and hit Ctrl+Scroll+Scroll during hang or during normal no-hang scenarios. Still no crash dump generated.

Is there a reliable way I can generate the crash dump on the hung x86 Win7 machine?

Thanks,
Chandra

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

> Hi,

I have a x86 Win7 machine, where I can repro a system hang (no mouse
and
keyboard) using one of our own drivers. The machine does not have a
COM or
IEEE 1394 port, so I can’t debug as the hang happens. When the machine
hangs,
it does not generate the crash dump. I played with all crash dump
generation
option under system settings, e.g., Setting “Complete Crash Dump” etc.
That
didn’t help.

I also tried setting CrashOnCtrlScroll registry value under service
“parameters” for both i8042prt.sys and kbhid.sys, and hit
Ctrl+Scroll+Scroll
during hang or during normal no-hang scenarios. Still no crash dump
generated.

Is there a reliable way I can generate the crash dump on the hung x86
Win7
machine?

I have never tried it myself but someone mentioned a way to synthesize a
windows dump file from a saved xen VM image. It may be possible to do
the same from VMWare/Hyper-V/VirtualPC/etc saved VM images.

If your driver is a filesystem filter or something then any of the above
should work. If it is for a PCI device then Xen can pass a PCI device to
a VM (not sure about the others). If it’s USB then you may need to pass
a PCI USB adapter through to the VM but it should still work.

James

James Harper wrote:

I have never tried it myself but someone mentioned a way to synthesize a
windows dump file from a saved xen VM image. It may be possible to do
the same from VMWare/Hyper-V/VirtualPC/etc saved VM images.

If your driver is a filesystem filter or something then any of the above
should work. If it is for a PCI device then Xen can pass a PCI device to
a VM (not sure about the others). If it’s USB then you may need to pass
a PCI USB adapter through to the VM but it should still work.

That’s a very interesting idea – and one that had not occurred to me –
but there is a problem. If his hang is truly being caused by a bus
issue or an interrupt storm, it’s going to hang the virtual host. The
saved image won’t be any good.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

>

James Harper wrote:
>
> I have never tried it myself but someone mentioned a way to
synthesize a
> windows dump file from a saved xen VM image. It may be possible to
do
> the same from VMWare/Hyper-V/VirtualPC/etc saved VM images.
>
> If your driver is a filesystem filter or something then any of the
above
> should work. If it is for a PCI device then Xen can pass a PCI
device to
> a VM (not sure about the others). If it’s USB then you may need to
pass
> a PCI USB adapter through to the VM but it should still work.

That’s a very interesting idea – and one that had not occurred to me

but there is a problem. If his hang is truly being caused by a bus
issue or an interrupt storm, it’s going to hang the virtual host. The
saved image won’t be any good.

True. It will be a clue that a bus issue or interrupt storm is what you
are dealing with though, as opposed to just spinning at some high IRQL
:slight_smile:

If you are using Xen and are handy with the Linux kernel/Xen hypervisor
(unlikely if you are a windows developer :p) you could easily add some
code to detect more than a certain number of interrupts being passed
through to the VM in a given period - an interrupt storm should be easy
to tell from regular usage, even with very course measurement.

Linux is more likely to give something useful in terms of error messages
too - delivering a message to a serial port or a text mode screen is
much easier and more reliable than popping up a message box or writing
something to disk.

One other thing I didn’t pick up at first - the OP says that their
machine doesn’t have a COM or 1394 interface… given the availability
of cheap hardware that does have a COM port this says to me that this is
some hardware device tied to the actual machine and so running under Xen
may not be an option either (linux support on laptops/embedded hardware
is much better than it used to be but it’s not always a given).

James

You should find the manual for the specific motherboard on the problematic machine. I know that the last machine that I built (a Supermicro) has a jumper for an NMI switch built into it.