recieving DRIVER_CORRUPTED_EXPOOL (c5)

NTFSD
1

Hi All,

I am receiving DRIVER_CORRUPTED_EXPOOL (c5), BAD_POOL_CALLER & IRQL_NOT_LESS_OR_EQUAL crashes with my driver. Frustratingly I am not able to make any thing out of the crash dump. Kindly suggest any way to go further. I am putting here what all I have observed yet.

Here is the dump.

DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is
caused by drivers that have corrupted the system pool. Run the driver
verifier against any new (or suspect) drivers, and if that doesn’t turn up
the culprit, then use gflags to enable special pool.
Arguments:
Arg1: 30003600, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: 8054b96d, address which referenced memory

Debugging Details:

BUGCHECK_STR: 0xC5_2

CURRENT_IRQL: 2

FAULTING_IP:
nt!ExDeferredFreePool+b4
8054b96d 8b10 mov edx,[eax]

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT

LAST_CONTROL_TRANSFER: from 8054b96d to 804dbaac

STACK_TEXT:
f78c6bb8 8054b96d badb0d00 891e1000 f78c6ba8 nt!KiTrap0E+0x238
f78c6c6c 8054bac7 00000000 8054b1e6 89194008 nt!ExDeferredFreePool+0xb4
f78c6cac 8054b3e3 89194008 00000000 f78c6cc8 nt!ExFreePoolWithTag+0x47f
f78c6cbc f7b547bb 89194008 f78c6ce4 f7b52481 nt!ExFreePool+0xf
f78c6cc8 f7b52481 f7b71e20 89194008 00000000 Ntfs!ExFreeToPagedLookasideList+0x1e
f78c6ce4 f7b52666 89194008 00000001 00000000 Ntfs!NtfsCleanupIrpContext+0x10d
f78c6cfc f7b7c414 89194008 00000000 00000000 Ntfs!NtfsCompleteRequest+0x35
f78c6d7c 804dd3b5 00000000 00000000 897da3c8 Ntfs!NtfsFspClose+0x1a5
f78c6dac 8056f128 00000000 00000000 00000000 nt!ExpWorkerThread+0xef
f78c6ddc 804e7781 804dd2f1 00000000 00000000 nt!PspSystemThreadStartup+0x34
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

We have a encryption/decryption minifilter driver & this bug is occurring only when I am trying to view a folder which contains files with really big names. I did review my IRP_MJ_DIRECTORY_CONTROL many times.

Few things I noticed further is every time Argument 3 value is 1 i.e. write operation. & some times it displays IRQL level as FF.

Any clues?

Aditya

2

I have enable verifier with special pool on my driver it still result into similar dump but surprisingly when I enable pool tracking with special pool; crash won’t appears. Currently trying with GFlags.

3

The error message essentially says what occurred; you’ve likely corrupted
the pool, overwritten past memory you shouldn’t have and so the pool walk
probably trapped.

Run “verifier” and enable special pool for starters, then reboot and rerun
the bug.

-----Original Message-----
From: xxxxx@gmail.com [mailto:xxxxx@gmail.com]
Sent: Tuesday, November 20, 2007 11:04 PM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] recieving DRIVER_CORRUPTED_EXPOOL (c5)

Hi All,

I am receiving DRIVER_CORRUPTED_EXPOOL (c5), BAD_POOL_CALLER &
IRQL_NOT_LESS_OR_EQUAL crashes with my driver. Frustratingly I am not able
to make any thing out of the crash dump. Kindly suggest any way to go
further. I am putting here what all I have observed yet.

Here is the dump.

DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at
an
interrupt request level (IRQL) that is too high. This is
caused by drivers that have corrupted the system pool. Run the driver
verifier against any new (or suspect) drivers, and if that doesn’t turn up
the culprit, then use gflags to enable special pool.
Arguments:
Arg1: 30003600, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: 8054b96d, address which referenced memory

Debugging Details:

BUGCHECK_STR: 0xC5_2

CURRENT_IRQL: 2

FAULTING_IP:
nt!ExDeferredFreePool+b4
8054b96d 8b10 mov edx,[eax]

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT

LAST_CONTROL_TRANSFER: from 8054b96d to 804dbaac

STACK_TEXT:
f78c6bb8 8054b96d badb0d00 891e1000 f78c6ba8 nt!KiTrap0E+0x238
f78c6c6c 8054bac7 00000000 8054b1e6 89194008 nt!ExDeferredFreePool+0xb4
f78c6cac 8054b3e3 89194008 00000000 f78c6cc8 nt!ExFreePoolWithTag+0x47f
f78c6cbc f7b547bb 89194008 f78c6ce4 f7b52481 nt!ExFreePool+0xf
f78c6cc8 f7b52481 f7b71e20 89194008 00000000
Ntfs!ExFreeToPagedLookasideList+0x1e
f78c6ce4 f7b52666 89194008 00000001 00000000
Ntfs!NtfsCleanupIrpContext+0x10d
f78c6cfc f7b7c414 89194008 00000000 00000000 Ntfs!NtfsCompleteRequest+0x35
f78c6d7c 804dd3b5 00000000 00000000 897da3c8 Ntfs!NtfsFspClose+0x1a5
f78c6dac 8056f128 00000000 00000000 00000000 nt!ExpWorkerThread+0xef
f78c6ddc 804e7781 804dd2f1 00000000 00000000 nt!PspSystemThreadStartup+0x34
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

We have a encryption/decryption minifilter driver & this bug is occurring
only when I am trying to view a folder which contains files with really big
names. I did review my IRP_MJ_DIRECTORY_CONTROL many times.

Few things I noticed further is every time Argument 3 value is 1 i.e. write
operation. & some times it displays IRQL level as FF.

Any clues?

Aditya

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@opferman.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

4

How similar of a trap? The next thing would be to walk the pool manually,
dump the headers and see if/where the pool headers become corrupt and find
out what memory is allocated before/after this location in the pool. The
pool may not have caught the bug if a free didn’t occur for it to check
before the problem occurs.

-----Original Message-----
From: xxxxx@gmail.com [mailto:xxxxx@gmail.com]
Sent: Tuesday, November 20, 2007 11:33 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] recieving DRIVER_CORRUPTED_EXPOOL (c5)

I have enable verifier with special pool on my driver it still result into
similar dump but surprisingly when I enable pool tracking with special
pool; crash won’t appears. Currently trying with GFlags.

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@opferman.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

5

You can dump and walk the stack manually to determine if these are correct
but perhaps starting with “89194008” based on the first dump you provided:

f78c6bb8 8054b96d badb0d00 891e1000 f78c6ba8 nt!KiTrap0E+0x238
f78c6c6c 8054bac7 00000000 8054b1e6 89194008 nt!ExDeferredFreePool+0xb4
f78c6cac 8054b3e3 89194008 00000000 f78c6cc8 nt!ExFreePoolWithTag+0x47f
f78c6cbc f7b547bb 89194008 f78c6ce4 f7b52481 nt!ExFreePool+0xf
f78c6cc8 f7b52481 f7b71e20 89194008 00000000
Ntfs!ExFreeToPagedLookasideList+0x1e

You can start with !pool and !poolval then just manually walk the pool.
There is an irp on the stack higher up you could also dump that and see what
is being cleaned up and perhaps if its an easy enough path set a break point
when that irp is dispatched, then on that location of the free and verify
it. If you can notice corruption and if its consistent you can even do
break on access at that location since this is easily reproducible as
opposed to something that is taking a week to reproduce.

-----Original Message-----
From: Toby Opferman [mailto:xxxxx@opferman.com]
Sent: Wednesday, November 21, 2007 12:24 AM
To: ‘Windows File Systems Devs Interest List’
Subject: RE: [ntfsd] recieving DRIVER_CORRUPTED_EXPOOL (c5)

How similar of a trap? The next thing would be to walk the pool manually,
dump the headers and see if/where the pool headers become corrupt and find
out what memory is allocated before/after this location in the pool. The
pool may not have caught the bug if a free didn’t occur for it to check
before the problem occurs.

-----Original Message-----
From: xxxxx@gmail.com [mailto:xxxxx@gmail.com]
Sent: Tuesday, November 20, 2007 11:33 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] recieving DRIVER_CORRUPTED_EXPOOL (c5)

I have enable verifier with special pool on my driver it still result into
similar dump but surprisingly when I enable pool tracking with special
pool; crash won’t appears. Currently trying with GFlags.

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@opferman.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

6

Thanks a lot Toby,

I tried what you suggested & got following details.

This time I am debugging it live & the call stack is

===============================================

DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is
caused by drivers that have corrupted the system pool. Run the driver
verifier against any new (or suspect) drivers, and if that doesn’t turn up
the culprit, then use gflags to enable special pool.
Arguments:
Arg1: 30003600, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: 8054b96d, address which referenced memory

Debugging Details:

BUGCHECK_STR: 0xC5_2

CURRENT_IRQL: 2

FAULTING_IP:
nt!ExDeferredFreePool+b4
8054b96d 8b10 mov edx,[eax]

DEFAULT_BUCKET_ID: DRIVER_FAULT

LAST_CONTROL_TRANSFER: from 8053129f to 804dda52

ChildEBP RetAddr Args to Child
a93d5d68 8053129f 00000003 a93d60c4 00000000 nt!RtlpBreakWithStatusInstruction
a93d5db4 80531d76 00000003 30003600 8054b96d nt!KiBugCheckDebugBreak+0x19
a93d6194 804dbaac 0000000a 30003600 00000002 nt!KeBugCheck2+0x574
a93d6194 8054b96d 0000000a 30003600 00000002 nt!KiTrap0E+0x238
a93d6268 8054bac7 00000000 00000000 892728c8 nt!ExDeferredFreePool+0xb4
a93d62a8 804e603a 892728c8 00000000 892728d8 nt!ExFreePoolWithTag+0x47f
a93d62c4 8057743b 892728c8 8926c010 00000000 nt!IopFreeIrp+0xed
a93d6300 8056778f 0026c028 00000000 8926c010 nt!IopDeleteFile+0x16d
a93d631c 804dc957 8926c028 00000000 00000e70 nt!ObpRemoveObjectRoutine+0xdf
a93d6334 80569a9e 890a0450 e1002e70 8947ada8 nt!ObfDereferenceObject+0x4c
a93d634c 805699bc e1002e70 8926c028 00000e70 nt!ObpCloseHandleTableEntry+0x155
a93d6394 80569a06 00000e70 00000000 00000000 nt!ObpCloseHandle+0x87
a93d63a8 804d899f 80000e70 a93d68f0 804de487 nt!NtClose+0x1d
a93d63a8 804de487 80000e70 a93d68f0 804de487 nt!KiFastCallEntry+0xfc
a93d6424 a9b2d8e8 80000e70 a93dd905 e37dfbea nt!ZwClose+0x11
a93d68f0 a9b2e9bb 89553ac0 e37df7d8 8914a000 Our Driver!CheckLogicalFileExist+0x242
a93d6bcc a9b2a46f e37df7d8 00000000 a93d6c67 Our Driver!GetVaultFileNameAtFilter+0x5fd
a93d709c a9b20283 896aeda4 a93d70fc a93d712c Our Driver!ProcessPreCreateOperation+0x465
a93d70dc bae84888 896aeda4 a93d70fc a93d712c Our Driver!HandlePreCreateOperation+0x25
a93d713c bae862a0 003d7180 896aed48 89227840 fltMgr!FltpPerformPreCallbacks+0x2d4

============================================================
Address being freed is 892728c8, as mentioned in call stack(Ist parameter to ExFreePoolWithTag). I did a !pool on this address & get following from windbg.

kd> !pool 892728c8
Pool page 892728c8 region is Nonpaged pool
89272000 size: 8c0 previous size: 0 (Free) Irp
*892728c0 size: 1c0 previous size: 8c0 (Free ) *Irp
Pooltag Irp : Io, IRP packets
89272a80 size: 1c0 previous size: 1c0 (Free) Irp
89272c40 size: 40 previous size: 1c0 (Allocated) TCPr
89272c80 size: 8 previous size: 40 (Free) Irp
89272c88 size: 30 previous size: 8 (Free ) TCiz
89272cb8 size: 40 previous size: 30 (Allocated) TCPr
89272cf8 size: 30 previous size: 40 (Allocated) Even (Protected)
89272d28 size: 1d8 previous size: 30 (Free) AfdE
89272f00 size: 30 previous size: 1d8 (Free ) TCi1
89272f30 size: a0 previous size: 30 (Free ) AfdC (Protected)
89272fd0 size: 30 previous size: a0 (Free ) Io

I also tried !poolval on 892728c8 & found this valid.

>find out what memory is allocated before/after this location in the pool.

I get the point & doing this currently.

>There is an irp on the stack higher up you could also dump that and see what is being cleaned up
I have used !Irp command on the parameter of IopFreeIrp but it says it is not a valid IRP. Any Clues?

>and perhaps if its an easy enough path set a break point when that irp is dispatched, then on that location of the free and verify it.
Actually this IRP is generated due to a call to FltCreateFile from OurDriver!CheckLogicalFileExist & this free IRP call is generated from FltClose API. I will try to analyze it further.

Just a question here : Is it possible by any way to jump into ZwCloseFile code? I know we do not have source code for this but just curious to knowif there is any way for same.

Is there any thing to be pointed from the !pool result?

Thanks Again
Aditya

7

I found it, a pointer is being written beyond its limit, due to some junk value used in it. Its a code error.

Thanks For Support