Memory leak analysis

I have a crash dump from user that they run out of virtual memory and
system hangs. After I analyze the memory dump, I don’t see any problem
with potential deadlock. Each process’s virtual size is also normal,
but if I do !vm, I get the following result:

!vm

*** Virtual Memory Usage ***
Physical Memory: 655239 ( 2620956 Kb)
Page File: ??\C:\pagefile.sys
Current: 3930112Kb Free Space: 3919636Kb
Minimum: 3930112Kb Maximum: 3930112Kb
Available Pages: 462198 ( 1848792 Kb)
ResAvail Pages: 570758 ( 2283032 Kb)
Modified Pages: 715 ( 2860 Kb)
NonPagedPool Usage: 60742 ( 242968 Kb)
NonPagedPool Max: 69377 ( 277508 Kb)
********** Excessive NonPaged Pool Usage ***** (memory leak)

The system is not enabled with pool tag, can anybody has better idea
about what further path I can move forward?
Paul

That doesn’t look like a virtual memory problem at all to me. Looks
like you
are using too much non-paged pool. Use !poolused, or !poolfind.

-Jeff

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Paul Mu
Sent: Wednesday, November 03, 2004 6:14 PM
To: Kernel Debugging Interest List
Subject: [windbg] Memory leak analysis

I have a crash dump from user that they run out of virtual memory and
system hangs. After I analyze the memory dump, I don’t see any problem
with potential deadlock. Each process’s virtual size is also normal,
but if I do !vm, I get the following result:

!vm

*** Virtual Memory Usage ***
Physical Memory: 655239 ( 2620956 Kb)
Page File: ??\C:\pagefile.sys
Current: 3930112Kb Free Space: 3919636Kb
Minimum: 3930112Kb Maximum: 3930112Kb
Available Pages: 462198 ( 1848792 Kb)
ResAvail Pages: 570758 ( 2283032 Kb)
Modified Pages: 715 ( 2860 Kb)
NonPagedPool Usage: 60742 ( 242968 Kb)
NonPagedPool Max: 69377 ( 277508 Kb)
********** Excessive NonPaged Pool Usage ***** (memory leak)

The system is not enabled with pool tag, can anybody has better idea
about what further path I can move forward? Paul


You are currently subscribed to windbg as: unknown lmsubst tag argument:
‘’ To unsubscribe send a blank email to xxxxx@lists.osr.com

***********************************************************************************
This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, received late or incomplete, or could contain viruses. The sender therefore does not accept liability for any error or omission in the contents of this message, which arises as a result of e-mail transmission. If verification is required, please request a hard-copy version from the sender.
***********************************************************************************

That’s probably right. As I said the system is not enabled with pool tag
so I can’t identify which driver uses that many non-paged memory. There
are several reputable filter drivers in the chain.

Paul

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Curless, Jeffrey
Sent: Thursday, November 04, 2004 7:36 AM
To: Kernel Debugging Interest List
Subject: RE: [windbg] Memory leak analysis

That doesn’t look like a virtual memory problem at all to me. Looks
like you
are using too much non-paged pool. Use !poolused, or !poolfind.

-Jeff

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Paul Mu
Sent: Wednesday, November 03, 2004 6:14 PM
To: Kernel Debugging Interest List
Subject: [windbg] Memory leak analysis

I have a crash dump from user that they run out of virtual memory and
system hangs. After I analyze the memory dump, I don’t see any problem
with potential deadlock. Each process’s virtual size is also normal,
but if I do !vm, I get the following result:

!vm

*** Virtual Memory Usage ***
Physical Memory: 655239 ( 2620956 Kb)
Page File: ??\C:\pagefile.sys
Current: 3930112Kb Free Space: 3919636Kb
Minimum: 3930112Kb Maximum: 3930112Kb
Available Pages: 462198 ( 1848792 Kb)
ResAvail Pages: 570758 ( 2283032 Kb)
Modified Pages: 715 ( 2860 Kb)
NonPagedPool Usage: 60742 ( 242968 Kb)
NonPagedPool Max: 69377 ( 277508 Kb)
********** Excessive NonPaged Pool Usage ***** (memory leak)

The system is not enabled with pool tag, can anybody has better idea
about what further path I can move forward? Paul


You are currently subscribed to windbg as: unknown lmsubst tag argument:
‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com

************************************************************************
***********
This message contains confidential information and is intended only for
the individual named. If you are not the named addressee you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system. E-mail transmission cannot be
guaranteed to be secured or error-free as information could be
intercepted, corrupted, lost, destroyed, received late or incomplete, or
could contain viruses. The sender therefore does not accept liability
for any error or omission in the contents of this message, which arises
as a result of e-mail transmission. If verification is required, please
request a hard-copy version from the sender.
************************************************************************
***********


You are currently subscribed to windbg as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Unfortunately, without pooltags you can’t do much debugging of a pool
leak. If this is a one-off dump and you MUST do debugging, then get
your pencil out and start looking at the individual pool pages to see if
these are small pool allocations (even without tagging, the small
allocations still store the tag value - they just don’t tally it up for
you)

This will not tell you the problem if it is large (PAGE_SIZE) leaks.
Non-Paged pool typically has a small region in low kernel memory (from
the large page where the OS was loaded/started) and then starts high in
memory, with the specific location varying by OS version (look at
nt!MmNonPagedPoolStart and MmNonPagedPoolStart0 for the lower region and
nt!MmNonPagedPoolExpansionStart and nt!MmNonPagedPoolEnd for the upper
memory region. You can do !pool on each page in the region and if it is
a small allocation you’ll get a nice dump of the individual pieces. If
this is not a small allocation page then you’ll get garbage.

Once you have gone through the various pages of the system (should be
around 60,742 of them given the size of nonpaged pool) with a bit of
list manipulation you can add things up. It might be faster to write a
debugger extension to do the walk-through and dump the data and then
post process the output.

If you have access to the system and can reproduce the problem, my
suggestion is to enable pool tags on the system and reproduce it. Then
the problem will drop out rather quickly. You can use a utility like
pooltag (in the DDK and on www.osronline.com in the downloads section)
to gather the information and enable pool tags as necessary.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the Next OSR File Systems Class October
18, 2004 in Silicon Valley!
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Paul Mu
Sent: Thursday, November 04, 2004 11:51 AM
To: Kernel Debugging Interest List
Subject: RE: [windbg] Memory leak analysis

That’s probably right. As I said the system is not enabled with pool tag
so I can’t identify which driver uses that many non-paged memory. There
are several reputable filter drivers in the chain.

Paul

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Curless, Jeffrey
Sent: Thursday, November 04, 2004 7:36 AM
To: Kernel Debugging Interest List
Subject: RE: [windbg] Memory leak analysis

That doesn’t look like a virtual memory problem at all to me. Looks
like you
are using too much non-paged pool. Use !poolused, or !poolfind.

-Jeff

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Paul Mu
Sent: Wednesday, November 03, 2004 6:14 PM
To: Kernel Debugging Interest List
Subject: [windbg] Memory leak analysis

I have a crash dump from user that they run out of virtual memory and
system hangs. After I analyze the memory dump, I don’t see any problem
with potential deadlock. Each process’s virtual size is also normal,
but if I do !vm, I get the following result:

!vm

*** Virtual Memory Usage ***
Physical Memory: 655239 ( 2620956 Kb)
Page File: ??\C:\pagefile.sys
Current: 3930112Kb Free Space: 3919636Kb
Minimum: 3930112Kb Maximum: 3930112Kb
Available Pages: 462198 ( 1848792 Kb)
ResAvail Pages: 570758 ( 2283032 Kb)
Modified Pages: 715 ( 2860 Kb)
NonPagedPool Usage: 60742 ( 242968 Kb)
NonPagedPool Max: 69377 ( 277508 Kb)
********** Excessive NonPaged Pool Usage ***** (memory leak)

The system is not enabled with pool tag, can anybody has better idea
about what further path I can move forward? Paul


You are currently subscribed to windbg as: unknown lmsubst tag argument:
‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com

************************************************************************
***********
This message contains confidential information and is intended only for
the individual named. If you are not the named addressee you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system. E-mail transmission cannot be
guaranteed to be secured or error-free as information could be
intercepted, corrupted, lost, destroyed, received late or incomplete, or
could contain viruses. The sender therefore does not accept liability
for any error or omission in the contents of this message, which arises
as a result of e-mail transmission. If verification is required, please
request a hard-copy version from the sender.
************************************************************************
***********


You are currently subscribed to windbg as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com


You are currently subscribed to windbg as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com