Crash Dump Analysis

Hi All,

I have some doubts in Crash Dump analysis.

  1. Kernel Dump and Full memory dump

I have taken kernel memory dump many times on my machine it comes usually
around 150KB. Are all extensions like !process 0 0, !irpfind etc are
applicable here, I see problem here even system has update symbols. Do I
need to take full dump?

    1. How to find reason for hang in the system? Where should we start
      to find out the cause?

a. If we find it is because of some lock the how to find which lock it
is?

b. If it is because of IRP or DPC hang then how to detect these?

    1. If there has been a thread which had overwritten some buffer due
      to which thread in other process creates a crash how to find who is
      original culprit and original buffer which got problem?(with or without
      driver verifier)
    1. How to detect memory leak in driver from dump?
    1. How to find current thread running?

Thanks and Regards

Sunil

Hi Sunil,

  1. A 150KB dump is going to be a triage/mini dump (called “Small Memory Dump” in the Advanced System Settings). A kernel memory dump should be at least a hundred MB on most systems. For details of the registry settings you can verify, see http://blogs.technet.com/b/askcore/archive/2012/09/12/windows-8-and-windows-server-2012-automatic-memory-dump.aspx

  2. This is pretty open ended question, since “hang in the system” can mean many different things. Let’s start with symbol loading. You should have srv*c:\symbols*http://msdl.microsoft.com/download/symbols in your symbol path, to search the symbol server and cache the symbols locally.

  • Check for running threads using !running -t (applicable if it’s a high CPU hang)
  • !vm, check for memory pressure/exhaustion
  • !locks will show owned ERESOURCEs, and the owner of those resources.
  • A DPC hang will eventually cause a 0x133 bugcheck, on Windows 8 and higher. On previous OS, if you have KD attached, a DebugBreak() will occur if a DPC timeout occurs.
  • !kdexts.swd will show the current state of software watchdog timers
  1. Finding the culprit of a buffer overrun, driver verifier is the proper way to catch the culprit in action, using special pool. With special pool enabled, the corrupted buffer will be identified by the 0xC4 bugcheck parameters.

  2. Pool Leaks can be identified via !poolused. Though, you will want to enable pool tagging with verifier to then be able to get callstacks of the allocations. This is another place where WPA is going to likely give you better results for finding root cause. See http://blogs.msdn.com/b/ntdebugging/archive/2012/11/30/troubleshooting-pool-leaks-part-7-windows-performance-toolkit.aspx

  3. !running -t

Regards,
Graham
Windows Reliability Team

For many types of performance issues / hangs, using WPA / XPerf to enable CSWITCH, DPC, ISR tracing is going to be more productive.

Thanx!! this is really helpful.

On Tue, May 21, 2013 at 4:13 PM, wrote:

> Hi Sunil,
>
> 1. A 150KB dump is going to be a triage/mini dump (called “Small Memory
> Dump” in the Advanced System Settings). A kernel memory dump should be at
> least a hundred MB on most systems. For details of the registry settings
> you can verify, see
> http://blogs.technet.com/b/askcore/archive/2012/09/12/windows-8-and-windows-server-2012-automatic-memory-dump.aspx
>
> 2. This is pretty open ended question, since “hang in the system” can mean
> many different things. Let’s start with symbol loading. You should have
> srvc:\symbolshttp://msdl.microsoft.com/download/symbols in your symbol
> path, to search the symbol server and cache the symbols locally.
>
> - Check for running threads using !running -t (applicable if it’s a high
> CPU hang)
> - !vm, check for memory pressure/exhaustion
> - !locks will show owned ERESOURCEs, and the owner of those resources.
> - A DPC hang will eventually cause a 0x133 bugcheck, on Windows 8 and
> higher. On previous OS, if you have KD attached, a DebugBreak() will occur
> if a DPC timeout occurs.
> - !kdexts.swd will show the current state of software watchdog timers
>
> 3. Finding the culprit of a buffer overrun, driver verifier is the proper
> way to catch the culprit in action, using special pool. With special pool
> enabled, the corrupted buffer will be identified by the 0xC4 bugcheck
> parameters.
>
> 4. Pool Leaks can be identified via !poolused. Though, you will want to
> enable pool tagging with verifier to then be able to get callstacks of the
> allocations. This is another place where WPA is going to likely give you
> better results for finding root cause. See
> http://blogs.msdn.com/b/ntdebugging/archive/2012/11/30/troubleshooting-pool-leaks-part-7-windows-performance-toolkit.aspx
>
> 5. !running -t
>
> Regards,
> Graham
> Windows Reliability Team
>
> For many types of performance issues / hangs, using WPA / XPerf to enable
> CSWITCH, DPC, ISR tracing is going to be more productive.
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

>update symbols. Do I need to take full dump?

Probably yes. What are the problems?

    1. How to find reason for hang in the system? Where should we start to find out the cause?

!locks, !process 0 7, !irp

    1. If there has been a thread which had overwritten some buffer due to which thread in other >process creates a crash how to find who is original culprit and original buffer which got problem?

Special pool can help.

    1. How to detect memory leak in driver from dump?

!poolused

    1. How to find current thread running?

!thread


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

On Wed, May 22, 2013 at 8:06 AM, Maxim S. Shatskih
wrote:

> >2. 2. How to find reason for hang in the system? Where should we start
> to find out the cause?
>
> !locks, !process 0 7, !irp

+ !stacks 2 if you know what you’re looking for.

Kris