ExAllocatePoolWithTag Fails to allocate!

NTFSD
1

Hello Everyone,

I am working on a minifilter driver where I am monitoring all IRP_MJ_CREATE & IRP_MJ_CLEANUP calls (pre & post)

I am using FLT_STREAMHANDLE_CONTEXT, to save some context info for only important calls that I need to monitor.

I have taken care of allocating and freeing required memory using the required functions like:
FltAllocateContext
FltReleaseContext
FltDeleteContext

Context allocation is done in the Post call of create and released in post call of clean up.

There is a que using the following functions:
ExAllocateFromNPagedLookasideList
ExFreeToNPagedLookasideList

The que is created in post cleanup and freed in the workitemroutine.

Also I am using the following functions to allocate local buffer when required:
ExAllocatePoolWithTag
ExFreePoolWithTag

I am allocating like 256 bytes buffer in every call and I make sure its freed at the end of the call.

The ExAllocatePoolWithTag call fails for a few calls. for example if there are 100 calls made may be 5 to 10 calls fail ramdomly.

I am loosing important calls due to this reason.

how can I make sure that the memory allocation never fails? or at least
how can I check if I am allocating too much memory?

The same behavior is on all machines that I am testing on and also all OS 32 & 64.

Thank you!

2

Silly question… you don’t have driver verifier running do you?

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Wednesday, October 17, 2012 10:51 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] ExAllocatePoolWithTag Fails to allocate!

Hello Everyone,

I am working on a minifilter driver where I am monitoring all IRP_MJ_CREATE & IRP_MJ_CLEANUP calls (pre & post)

I am using FLT_STREAMHANDLE_CONTEXT, to save some context info for only important calls that I need to monitor.

I have taken care of allocating and freeing required memory using the required functions like:
FltAllocateContext
FltReleaseContext
FltDeleteContext

Context allocation is done in the Post call of create and released in post call of clean up.

There is a que using the following functions:
ExAllocateFromNPagedLookasideList
ExFreeToNPagedLookasideList

The que is created in post cleanup and freed in the workitemroutine.

Also I am using the following functions to allocate local buffer when required:
ExAllocatePoolWithTag
ExFreePoolWithTag

I am allocating like 256 bytes buffer in every call and I make sure its freed at the end of the call.

The ExAllocatePoolWithTag call fails for a few calls. for example if there are 100 calls made may be 5 to 10 calls fail ramdomly.

I am loosing important calls due to this reason.

how can I make sure that the memory allocation never fails? or at least how can I check if I am allocating too much memory?

The same behavior is on all machines that I am testing on and also all OS 32 & 64.

Thank you!

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

3

As a matter of fact I have! verifier does not report any issue even with low res option!

4

Indeed a silly question! :wink:

Its working just as expected!

I had the verifier on with low res option :slight_smile: this failed a few allocs as per design!

Thank you Jeff for reminding me about the verifier!

5

> The ExAllocatePoolWithTag call fails for a few calls. for example if there are 100 calls made may be

5 to 10 calls fail ramdomly.

Do you have Verifier/Low Resources Simulation on?

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

6

It is expected that you will write your driver with the assumption that
any allocation call can fail at any time for any reason whatsoever, and
that your driver will either continue to function correstly or will fail
“gracefuuly”. Thereis absolutely NO way to “guarantee” that every
allocation will succex which is why you have to deal with these failures.
One way might beto allocate a lookaside list and pre-populate it. Set
the “depth” parameter to guarantee that the most common usage patterns
will always find sonething in the lookaside list. Then assume that this,
too, can fail at times. If you ever, for any reason, assume that every
allocation call will succeed, you are living in a fantasy world. Careful
design and engineering are required to increase robustness under these
failures, but don’t hope for a failure-proof solution. The lookaside-list
solution merely reduces the probability of failure. But if done right, it
might reduce the chane of failure to be sufficiently close to 0 that in
practice it no longer is a serious problem.
joe

Hello Everyone,

I am working on a minifilter driver where I am monitoring all
IRP_MJ_CREATE & IRP_MJ_CLEANUP calls (pre & post)

I am using FLT_STREAMHANDLE_CONTEXT, to save some context info for only
important calls that I need to monitor.

I have taken care of allocating and freeing required memory using the
required functions like:
FltAllocateContext
FltReleaseContext
FltDeleteContext

Context allocation is done in the Post call of create and released in post
call of clean up.

There is a que using the following functions:
ExAllocateFromNPagedLookasideList
ExFreeToNPagedLookasideList

The que is created in post cleanup and freed in the workitemroutine.

Also I am using the following functions to allocate local buffer when
required:
ExAllocatePoolWithTag
ExFreePoolWithTag

I am allocating like 256 bytes buffer in every call and I make sure its
freed at the end of the call.

The ExAllocatePoolWithTag call fails for a few calls. for example if there
are 100 calls made may be 5 to 10 calls fail ramdomly.

I am loosing important calls due to this reason.

how can I make sure that the memory allocation never fails? or at least
how can I check if I am allocating too much memory?

The same behavior is on all machines that I am testing on and also all OS
32 & 64.

Thank you!

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

7

>

It is expected that you will write your driver with the assumption that any
allocation call can fail at any time for any reason whatsoever, and that your
driver will either continue to function correstly or will fail “gracefuuly”.
Thereis absolutely NO way to “guarantee” that every allocation will succex
which is why you have to deal with these failures.
One way might beto allocate a lookaside list and pre-populate it. Set the
“depth” parameter to guarantee that the most common usage patterns will
always find sonething in the lookaside list. Then assume that this, too, can fail
at times. If you ever, for any reason, assume that every allocation call will
succeed, you are living in a fantasy world. Careful design and engineering are
required to increase robustness under these failures, but don’t hope for a
failure-proof solution. The lookaside-list solution merely reduces the
probability of failure. But if done right, it might reduce the chane of failure to
be sufficiently close to 0 that in practice it no longer is a serious problem.

Watch out for leaks too. In an operation that relies on allocating (say) 3 separate resources, a failure of allocating the 3rd requires releasing the first two. I recently found a leak in one of my drivers that I’d somehow missed that meant in the very rare case of allocating the 3rd resource I released the first two resources in the wrong order, which leaked a resource and then ultimately crashed much later on. Because out-of-resource errors are rare this sort of problem can lay dormant for a long time and only comes out when the system is pushed to its limits.

James

8

Very good point, and it is a common error.
joe

>
> It is expected that you will write your driver with the assumption that
> any
> allocation call can fail at any time for any reason whatsoever, and that
> your
> driver will either continue to function correstly or will fail
> “gracefuuly”.
> Thereis absolutely NO way to “guarantee” that every allocation will
> succex
> which is why you have to deal with these failures.
> One way might beto allocate a lookaside list and pre-populate it. Set
> the
> “depth” parameter to guarantee that the most common usage patterns will
> always find sonething in the lookaside list. Then assume that this, too,
> can fail
> at times. If you ever, for any reason, assume that every allocation
> call will
> succeed, you are living in a fantasy world. Careful design and
> engineering are
> required to increase robustness under these failures, but don’t hope for
> a
> failure-proof solution. The lookaside-list solution merely reduces the
> probability of failure. But if done right, it might reduce the chane of
> failure to
> be sufficiently close to 0 that in practice it no longer is a serious
> problem.

Watch out for leaks too. In an operation that relies on allocating (say) 3
separate resources, a failure of allocating the 3rd requires releasing the
first two. I recently found a leak in one of my drivers that I’d somehow
missed that meant in the very rare case of allocating the 3rd resource I
released the first two resources in the wrong order, which leaked a
resource and then ultimately crashed much later on. Because
out-of-resource errors are rare this sort of problem can lay dormant for a
long time and only comes out when the system is pushed to its limits.

James

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

9

Prefast usually is (was?) very good at catching problem like this.

On Thu, Oct 18, 2012 at 6:42 PM, James Harper > wrote:
>
> Watch out for leaks too. In an operation that relies on allocating (say) 3
> separate resources, a failure of allocating the 3rd requires releasing the
> first two. I recently found a leak in one of my drivers that I’d somehow
> missed that meant in the very rare case of allocating the 3rd resource I
> released the first two resources in the wrong order, which leaked a
> resource and then ultimately crashed much later on. Because out-of-resource
> errors are rare this sort of problem can lay dormant for a long time and
> only comes out when the system is pushed to its limits.
>
> James
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>