ExDeleteNPagedLookasideList -- XP vs W2K3SP1

NTDEV
1

Driver Verifier on Win2003Server SP1 is tossing a fatal error about failure to
release non-paged pool in my driver unload routine. Verifier on XP-SP2 sees no
such failure, but since it is claimed that Verifier on W2K3 is “better”, I spent
some time trying to track down the problem in my driver.

The first thing I noticed is that the windbg cmd: !verifier 3 mydriver.sys shows
a legitimate “unreleased NPP block” size, but when I look at the location that
allocation was made from, it is something nonsensical like:

mydriver!ExAllocateFromNPagedLookasideList+0x51
[c:\winddk\3790.1830\inc\ddk\wnet\ntddk.h @ 15317]:

This leads me to believe that the allocation info that verifier is capturing is
not the same as the XPSP2 verifier (which gives me the location in my driver
where the allocation occurs, not the address of the allocation routine).

In any case, I stepped through my driver’s code, with a bp on any type of pool
allocation, and found only one ExInitializeNPagedLookasideList call, several
allocations and frees from the LAL, and a ExDeleteNPagedLookasideList call in
the unload. It is not guaranteed that the allocations from the LAL have a
matching number of frees to the LAL, however.

Verifier’s “LAL leak” suggests that documentation on the LookasideList stuff may
not be correct with regard to the behavior of ExDeleteNPagedLookasideList. It
appears that the docs do not care whether or not ExFreeToNPagedLookasideList was
used to return any outstanding LAL allocations before deleting the LAL itself.

While good programming practice would be to free any pointers to LAL blocks that
were allocated from the LAL, after deleting the LAL, it is sufficient to just
avoid using any outstanding pointers to LAL blocks afte the delete. The reason I
believe this is that the docs (WXP-SP2 and W2K3-SP1 DDKs) both state only that
“Drivers must always use explicitly free any lookaside lists they create before
unloading”, and don’t even imply that allocated LAL block pointer must be freed
first.

Is this new difference in behavior between Win2003ServerSP1 and earlier OS LAL
implementation just a major flaw in the docs??? I’m also interested in whether
or not verifier’s failure to save the address of my driver’s LAL allocation call
is a new behavior or just an “issue”.

Suggestions, comments …

Jerry Schneider

2

It has always been apparent to me from reading the docs that

  1. all allocations made out of the lookaside must be freed back to the
    lookaside
  2. all allocations made out of the lookaside must be freed back to the
    lookaside before freeing the lookaside itself.

Those are the rules I follow and I have never had any problems. As for
the 2nd issue, the verifier captures the calling address of
ExAllocatePool when tracking the allocation. Since
ExAllocateFromNPagedLookasideList is a real function, the return address
is within that function and not the function that called
ExAllocateFromNPagedLookasideList.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jerry Schneider
Sent: Thursday, May 12, 2005 11:03 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] ExDeleteNPagedLookasideList – XP vs W2K3SP1

Driver Verifier on Win2003Server SP1 is tossing a fatal error about
failure to release non-paged pool in my driver unload routine. Verifier
on XP-SP2 sees no such failure, but since it is claimed that Verifier on
W2K3 is “better”, I spent some time trying to track down the problem in
my driver.

The first thing I noticed is that the windbg cmd: !verifier 3
mydriver.sys shows a legitimate “unreleased NPP block” size, but when I
look at the location that allocation was made from, it is something
nonsensical like:

mydriver!ExAllocateFromNPagedLookasideList+0x51
[c:\winddk\3790.1830\inc\ddk\wnet\ntddk.h @ 15317]:

This leads me to believe that the allocation info that verifier is
capturing is not the same as the XPSP2 verifier (which gives me the
location in my driver where the allocation occurs, not the address of
the allocation routine).

In any case, I stepped through my driver’s code, with a bp on any type
of pool allocation, and found only one ExInitializeNPagedLookasideList
call, several allocations and frees from the LAL, and a
ExDeleteNPagedLookasideList call in the unload. It is not guaranteed
that the allocations from the LAL have a matching number of frees to the
LAL, however.

Verifier’s “LAL leak” suggests that documentation on the LookasideList
stuff may not be correct with regard to the behavior of
ExDeleteNPagedLookasideList. It appears that the docs do not care
whether or not ExFreeToNPagedLookasideList was used to return any
outstanding LAL allocations before deleting the LAL itself.

While good programming practice would be to free any pointers to LAL
blocks that were allocated from the LAL, after deleting the LAL, it is
sufficient to just avoid using any outstanding pointers to LAL blocks
afte the delete. The reason I believe this is that the docs (WXP-SP2 and
W2K3-SP1 DDKs) both state only that “Drivers must always use explicitly
free any lookaside lists they create before unloading”, and don’t even
imply that allocated LAL block pointer must be freed first.

Is this new difference in behavior between Win2003ServerSP1 and earlier
OS LAL implementation just a major flaw in the docs??? I’m also
interested in whether or not verifier’s failure to save the address of
my driver’s LAL allocation call is a new behavior or just an “issue”.

Suggestions, comments …

Jerry Schneider

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

3

Doron Holan wrote:

It has always been apparent to me from reading the docs that

  1. all allocations made out of the lookaside must be freed back to the
    lookaside
  2. all allocations made out of the lookaside must be freed back to the
    lookaside before freeing the lookaside itself.

Those are the rules I follow and I have never had any problems.

Likewise, I follow these just because it is good programming practice. In
the current case, however, I’m trying to gracefully recover from a try/
except in a big block of code that happens to have one of the LAL blocks
allocated “somewhere inside the exception-tossing block of code” and it’s
a lot of work to locate the actual pointer to free it back to the LAL.

When I read the docs to see the consequences of this, the following seems
to allow deleting the LAL even with an outstanding allocation:

"The user of the lookaside list can allocate and free such entries dynamically
on an as needed basis until it calls ExDeleteNPagedLookasideList, which releases
any outstanding entries in the list before it clears the system state for the
given lookaside list and returns control.
"

The reference to “outstanding entries” is sufficiently vague that when my XP
driver unloaded successfully after an exception, I assumed my interpretation was
correct. Me thinks the docs need a bit more clarification, since I now see that
it possibly means that “outstanding entries” refer to the internal LAL entries
that are being held in the LAL for reallocation rather than being released to Pool.

Maybe 2003Server’s Verifier is better than XP’s, since it caught the leak. OTOH,
XP may used a different implementation for LALs…

As for
the 2nd issue, the verifier captures the calling address of
ExAllocatePool when tracking the allocation. Since
ExAllocateFromNPagedLookasideList is a real function, the return address
is within that function and not the function that called
ExAllocateFromNPagedLookasideList.

Somehow I thought ExAllocateFromNPagedLaL was another of those imbrogliated
macros that the DDK uses to optimize that type of allocation. I shoulda read
the source…

Thanks

4

“which releases any outstanding entries *** in the list *** before it
clears …”

The lookside list is a list of blobs avaialble for allocation. When
allocated they are no longer in the list. It’s not a like a heap/slab
allocator where freeing the heap/slab by definition gets rid of any
allocated chunks.

If you could, please provide this feedback through the link on the
documentation page.

-p

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jerry Schneider
Sent: Thursday, May 12, 2005 11:37 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] ExDeleteNPagedLookasideList – XP vs W2K3SP1

Doron Holan wrote:

It has always been apparent to me from reading the docs that

  1. all allocations made out of the lookaside must be freed back to
    the lookaside
  2. all allocations made out of the lookaside must be freed back to
    the lookaside before freeing the lookaside itself.

Those are the rules I follow and I have never had any problems.

Likewise, I follow these just because it is good programming practice.
In the current case, however, I’m trying to gracefully recover from a
try/ except in a big block of code that happens to have one of the LAL
blocks allocated “somewhere inside the exception-tossing block of code”
and it’s a lot of work to locate the actual pointer to free it back to
the LAL.

When I read the docs to see the consequences of this, the following
seems to allow deleting the LAL even with an outstanding allocation:

"The user of the lookaside list can allocate and free such entries
dynamically on an as needed basis until it calls
ExDeleteNPagedLookasideList, which releases any outstanding entries in
the list before it clears the system state for the given lookaside list
and returns control.
"

The reference to “outstanding entries” is sufficiently vague that when
my XP driver unloaded successfully after an exception, I assumed my
interpretation was correct. Me thinks the docs need a bit more
clarification, since I now see that it possibly means that “outstanding
entries” refer to the internal LAL entries that are being held in the
LAL for reallocation rather than being released to Pool.

Maybe 2003Server’s Verifier is better than XP’s, since it caught the
leak. OTOH, XP may used a different implementation for LALs…

As for
the 2nd issue, the verifier captures the calling address of
ExAllocatePool when tracking the allocation. Since
ExAllocateFromNPagedLookasideList is a real function, the return
address
is within that function and not the function that called
ExAllocateFromNPagedLookasideList.

Somehow I thought ExAllocateFromNPagedLaL was another of those
imbrogliated
macros that the DDK uses to optimize that type of allocation. I shoulda
read
the source…

Thanks

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

5

Jerry Schneider wrote:

Doron Holan wrote:

> It has always been apparent to me from reading the docs that
> 1) all allocations made out of the lookaside must be freed back to the
> lookaside
> 2) all allocations made out of the lookaside must be freed back to the
> lookaside before freeing the lookaside itself.
>
> Those are the rules I follow and I have never had any problems.

Likewise, I follow these just because it is good programming practice.

It’s more than “good programming practice”. A driver uses a lookaside
list to reuse as much as possible
the blocks of memory it allocates. This will reduce the memory
fragmentation and will generally speed up the allocation.
Failing to release the memory using ExFreeToNPagedLookasideList() is
equivalent to leaking memory.

In
the current case, however, I’m trying to gracefully recover from a try/
except in a big block of code that happens to have one of the LAL blocks
allocated “somewhere inside the exception-tossing block of code” and it’s
a lot of work to locate the actual pointer to free it back to the LAL.

When I read the docs to see the consequences of this, the following seems
to allow deleting the LAL even with an outstanding allocation:

"The user of the lookaside list can allocate and free such entries
dynamically on an as needed basis until it calls
ExDeleteNPagedLookasideList, which releases any outstanding entries in
the list before it clears the system state for the given lookaside
list and returns control.
"

The reference to “outstanding entries” is sufficiently vague that when
my XP driver unloaded successfully after an exception, I assumed my
interpretation was correct. Me thinks the docs need a bit more
clarification, since I now see that it possibly means that
“outstanding entries” refer to the internal LAL entries that are being
held in the LAL for reallocation rather than being released to Pool.

Maybe 2003Server’s Verifier is better than XP’s, since it caught the
leak. OTOH, XP may used a different implementation for LALs…

> As for
> the 2nd issue, the verifier captures the calling address of
> ExAllocatePool when tracking the allocation. Since
> ExAllocateFromNPagedLookasideList is a real function, the return address
> is within that function and not the function that called
> ExAllocateFromNPagedLookasideList.

Somehow I thought ExAllocateFromNPagedLaL was another of those
imbrogliated macros that the DDK uses to optimize that type of
allocation. I shoulda read the source…

Thanks

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@bitdefender.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

–
Ignorance more frequently begets confidence than does knowledge.
— Charles Darwin

–
This message was scanned for spam and viruses by BitDefender.
For more information please visit http://linux.bitdefender.com/

6

Andrei Zlate-Podani wrote:

It’s more than “good programming practice”. A driver uses a lookaside
list to reuse as much as possible
the blocks of memory it allocates. This will reduce the memory
fragmentation and will generally speed up the allocation.
Failing to release the memory using ExFreeToNPagedLookasideList() is
equivalent to leaking memory.

Actually, until recently, the LookasideList maintained its own list
of allocated pool blocks that it could free when the LAL was deleted.
Each allocation from the LAL was just a pointer to the next available
block in the internal LAL list, but the list did not lose track of
the block. Thus, deleting the LAL resulted in freeing each of the
blocks in the internal list, whether or not an “outstanding” pointer
to a block existed or not.

Recently (W2K3 Gold or SP1), the behavior was changed, probably for
efficiency, so that the LAL actually gives ownership of each allocated
block to the caller, meaning that deleting the LAL no longer could
free all of the LAL’s blocks – only the ones that were unallocated.

In my special case where an outstanding LAL block pointer “disappeared”
inside a try/except block, I just keep a copy of the pointers as they
are allocated, and can return any outstanding blocks to the LAL before
attempting to unload the driver.

My original question was asking two things – did the behavior actually
change from XP to W2K3, and if the docs were ambiguous about the
behavior. Thanks to all of the comments (and some debugging on
both OSes), I have a solution that works on all (current) OSes.

Jerry

7

The LAL never *EVER* freed memory for you that was not freed back to the
LAL. Think about it. Once you called
ExAllocateFromNPagedLookasideList, there is nowhere for the LAL to store
the next pointer to the next block. As such, it could never maintain
list consistency. The behavior of a LAL didn’t change, the OS just got
better at detecting your leak.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jerry Schneider
Sent: Friday, May 13, 2005 8:09 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] ExDeleteNPagedLookasideList – XP vs W2K3SP1

Andrei Zlate-Podani wrote:

It’s more than “good programming practice”. A driver uses a lookaside
list to reuse as much as possible
the blocks of memory it allocates. This will reduce the memory
fragmentation and will generally speed up the allocation.
Failing to release the memory using ExFreeToNPagedLookasideList() is
equivalent to leaking memory.

Actually, until recently, the LookasideList maintained its own list
of allocated pool blocks that it could free when the LAL was deleted.
Each allocation from the LAL was just a pointer to the next available
block in the internal LAL list, but the list did not lose track of
the block. Thus, deleting the LAL resulted in freeing each of the
blocks in the internal list, whether or not an “outstanding” pointer
to a block existed or not.

Recently (W2K3 Gold or SP1), the behavior was changed, probably for
efficiency, so that the LAL actually gives ownership of each allocated
block to the caller, meaning that deleting the LAL no longer could
free all of the LAL’s blocks – only the ones that were unallocated.

In my special case where an outstanding LAL block pointer “disappeared”
inside a try/except block, I just keep a copy of the pointers as they
are allocated, and can return any outstanding blocks to the LAL before
attempting to unload the driver.

My original question was asking two things – did the behavior actually
change from XP to W2K3, and if the docs were ambiguous about the
behavior. Thanks to all of the comments (and some debugging on
both OSes), I have a solution that works on all (current) OSes.

Jerry

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

8

Doron Holan wrote:

The LAL never *EVER* freed memory for you that was not freed back to the
LAL. Think about it. Once you called
ExAllocateFromNPagedLookasideList, there is nowhere for the LAL to store
the next pointer to the next block. As such, it could never maintain
list consistency. The behavior of a LAL didn’t change, the OS just got
better at detecting your leak.

My bad… I was “sure” that none of my tagged pool was being leaked
when my driver unloaded on XP, and since I initialized my LALs with
my tag, I assumed that LALs must be freeing any outstanding memory.
Chances are that subtle timing between my driver’s threads changed
between XP and W2K3, and I was freeing all allocated LAL blocks before
deleting the LAL. Sorry to have stated something as a fact when it
was a deduction (Plato, I appologize).

Jerry

9

> In my special case where an outstanding LAL block pointer “disappeared”

inside a try/except block, I just keep a copy of the pointers as they
are allocated, and can return any outstanding blocks to the LAL before
attempting to unload the driver.

Using try/except to work out your own bugs is usually a bad idea. Fix the bugs
instead. :slight_smile:

Also note that try/catch in the kernel has limitations about in what context
can it be used.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

10

Maxim S. Shatskih wrote:

>In my special case where an outstanding LAL block pointer “disappeared”
>inside a try/except block,

Using try/except to work out your own bugs is usually a bad idea. Fix the bugs
instead. :slight_smile:

In this case, my driver is “self-destruct” code injected into a particular style
of malware so that the malware’s driver is “neutered” after it tries to hook
certain services. It’s the malware that is the “bug”. ;->

Also note that try/catch in the kernel has limitations about in what context
can it be used.
Hmmm. I’m using it in an arbitrary context from user-mode ntdll.dll calls into
the kernel. I haven’t seen anything to suggest limitations in this type of
usage – can you point me to the limitations you are speaking of?

Thanks

Jerry

11

> In this case, my driver is “self-destruct” code injected into a particular
style

of malware so that the malware’s driver is “neutered” after it tries to hook
certain services. It’s the malware that is the “bug”. ;->

Oh, I see :slight_smile:

Hmmm. I’m using it in an arbitrary context from user-mode ntdll.dll calls
into
the kernel. I haven’t seen anything to suggest limitations in this type of
usage – can you point me to the limitations you are speaking of?

DISPATCH_LEVEL. The IRQL limitations.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

12

Jerry Schneider wrote:

Andrei Zlate-Podani wrote:

> It’s more than “good programming practice”. A driver uses a lookaside
> list to reuse as much as possible
> the blocks of memory it allocates. This will reduce the memory
> fragmentation and will generally speed up the allocation.
> Failing to release the memory using ExFreeToNPagedLookasideList() is
> equivalent to leaking memory.

Actually, until recently, the LookasideList maintained its own list
of allocated pool blocks that it could free when the LAL was deleted.
Each allocation from the LAL was just a pointer to the next available
block in the internal LAL list, but the list did not lose track of
the block. Thus, deleting the LAL resulted in freeing each of the
blocks in the internal list, whether or not an “outstanding” pointer
to a block existed or not.

Even if it was true, that does not contradict what I was saying. Think
of the heap the user-mode apps get.
You can leak memory even in java (although some would contradict me :slight_smile: ).

Recently (W2K3 Gold or SP1), the behavior was changed, probably for
efficiency, so that the LAL actually gives ownership of each allocated
block to the caller, meaning that deleting the LAL no longer could
free all of the LAL’s blocks – only the ones that were unallocated.

In my special case where an outstanding LAL block pointer “disappeared”
inside a try/except block, I just keep a copy of the pointers as they
are allocated, and can return any outstanding blocks to the LAL before
attempting to unload the driver.

My original question was asking two things – did the behavior actually
change from XP to W2K3, and if the docs were ambiguous about the
behavior. Thanks to all of the comments (and some debugging on
both OSes), I have a solution that works on all (current) OSes.

Jerry

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@bitdefender.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

–
Ignorance more frequently begets confidence than does knowledge.
— Charles Darwin

–
This message was scanned for spam and viruses by BitDefender.
For more information please visit http://linux.bitdefender.com/