Paged allocation leakage with verifier on

NTDEV
1

Hi folks,

Verifier bugchecks my driver when the driver is being unloaded. And the Verifier says there are allocations not freed in our driver.

1: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Parameter 1 = 0x1000 .. 0x1020 - deadlock verifier error codes.
Typically the code is 0x1001 (deadlock detected) and you can
issue a '!deadlock' KD command to get more information.
Arguments:
Arg1: 00000062, A driver has forgotten to free its pool allocations prior to unloading.
Arg2: 8713cabc, name of the driver having the issue.
Arg3: 8713b700, verifier internal structure with driver information.
Arg4: 00000001, total # of (paged+nonpaged) allocations that weren't freed.
Type !verifier 3 drivername.sys for info on the allocations
that were leaked that caused the bugcheck.

Debugging Details:

BUGCHECK_STR: 0xc4_62

IMAGE_NAME: mydriver.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4d7db948

MODULE_NAME: mydriver

FAULTING_MODULE: 93d71000 mydriver

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

PROCESS_NAME: System

CURRENT_IRQL: 2

LAST_CONTROL_TRANSFER: from 82b52f1f to 828fadfc

STACK_TEXT:
87b27a90 82b52f1f 000000c4 00000062 8713cabc nt!KeBugCheckEx+0x1e
87b27ab0 82b57607 8713cabc 8713b700 93d71000 nt!VerifierBugCheckIfAppropriate+0x30
87b27ac0 8282aeaa 8713ca60 8295dec8 8295dec8 nt!VfPoolCheckForLeaks+0x33
87b27afc 829af6b4 8713ca60 93d71000 40000000 nt!VfTargetDriversRemove+0x66
87b27b10 829af34d 829667e0 851eca70 00000000 nt!VfDriverUnloadImage+0x5e
87b27b48 829b05a2 8713ca60 ffffffff 00000000 nt!MiUnloadSystemImage+0x1c6
87b27b6c 82ad9a2b 8713ca60 851eeeb0 8714edf8 nt!MmUnloadSystemImage+0x36
87b27b84 82a3fa98 8714ee10 8714ee10 8714edf8 nt!IopDeleteDriver+0x38
87b27b9c 82887080 00000000 8245e800 8245e750 nt!ObpRemoveObjectRoutine+0x59
87b27bb0 82886ff0 8714ee10 829d7cd2 851eef78 nt!ObfDereferenceObjectWithTag+0x88
87b27bb8 829d7cd2 851eef78 8245e738 8245e750 nt!ObfDereferenceObject+0xd
87b27bcc 82a3fa98 8245e750 8245e750 8245e738 nt!IopDeleteDevice+0x4e
87b27be4 82887080 00000000 82983e80 851e60d8 nt!ObpRemoveObjectRoutine+0x59
87b27bf8 82886ff0 8245e750 829d0384 a6cfacd0 nt!ObfDereferenceObjectWithTag+0x88
87b27c00 829d0384 a6cfacd0 a69ed8f8 00000000 nt!ObfDereferenceObject+0xd
87b27c1c 829d130d a69ed8f8 a9e512d0 a9e512b0 nt!PnpUnlinkDeviceRemovalRelations+0xe0
87b27cc4 829d3226 87b27cf4 00000000 a9e512b0 nt!PnpProcessQueryRemoveAndEject+0xa05
87b27cdc 829d4d6e 00000000 90488c18 851eca70 nt!PnpProcessTargetDeviceEvent+0x38
87b27d00 8288c03b 90488c18 00000000 851eca70 nt!PnpDeviceEventWorker+0x216
87b27d50 82a2c9df 00000001 8f61c205 00000000 nt!ExpWorkerThread+0x10d
87b27d90 828de1d9 8288bf2e 00000001 00000000 nt!PspSystemThreadStartup+0x9e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19

STACK_COMMAND: kb

FOLLOWUP_NAME: MachineOwner

FAILURE_BUCKET_ID: 0xc4_62_VRF_IMAGE_mydriver.sys

BUCKET_ID: 0xc4_62_VRF_IMAGE_mydriver.sys

Followup: MachineOwner

Then I type "!verifier 3 mydriver.sys". It says the leaked pool is at paged pool, but it doesn't supply the address of the leadked memory. Please look at below:

1: kd> !verifier 3 mydriver.sys

Verify Level 9bb ... enabled options are:
Special pool
Special irql
All pool allocations checked on unload
Io subsystem checking enabled
Deadlock detection enabled
DMA checking enabled
Security checks enabled
Miscellaneous checks enabled

Summary of All Verifier Statistics

RaiseIrqls 0x0
AcquireSpinLocks 0x2032
Synch Executions 0x0
Trims 0x10

Pool Allocations Attempted 0x53ec5
Pool Allocations Succeeded 0x53ec5
Pool Allocations Succeeded SpecialPool 0x53ec5
Pool Allocations With NO TAG 0x0
Pool Allocations Failed 0x0
Resource Allocations Failed Deliberately 0x0

Current paged pool allocations 0x1 for 00000030 bytes
Peak paged pool allocations 0x7 for 00000600 bytes
Current nonpaged pool allocations 0x0 for 00000000 bytes
Peak nonpaged pool allocations 0x25 for 0000F744 bytes

Driver Verification List

Entry State NonPagedPool PagedPool Module

ERROR: BaseAddress is NULL for suspect drivers list entry at 850a4b28.

I don't know what module the leaked pool is at.

What should I do to find out the leakage function in my codes?

Any help is appreciated!!!

Joseph

2

ERROR: BaseAddress is NULL for suspect drivers list entry at 850a4b28.

check what’s in 850a4b28.

2011/3/17

> Hi folks,
>
> Verifier bugchecks my driver when the driver is being unloaded. And the
> Verifier says there are allocations not freed in our driver.
>
> 1: kd> !analyze -v
>
> *****
>
>
> * Bugcheck Analysis
>
>
>
>
>
>
> DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
> A device driver attempting to corrupt the system has been caught. This is
> because the driver was specified in the registry as being suspect (by the
> administrator) and the kernel has enabled substantial checking of this
> driver.
> If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
> will
> be among the most commonly seen crashes.
> Parameter 1 = 0x1000 … 0x1020 - deadlock verifier error codes.
> Typically the code is 0x1001 (deadlock detected) and you can
> issue a ‘!deadlock’ KD command to get more information.
> Arguments:
> Arg1: 00000062, A driver has forgotten to free its pool allocations prior
> to unloading.
> Arg2: 8713cabc, name of the driver having the issue.
> Arg3: 8713b700, verifier internal structure with driver information.
> Arg4: 00000001, total # of (paged+nonpaged) allocations that weren’t freed.
> Type !verifier 3 drivername.sys for info on the allocations
> that were leaked that caused the bugcheck.
>
> Debugging Details:
> ------------------
>
> BUGCHECK_STR: 0xc4_62
>
> IMAGE_NAME: mydriver.sys
>
> DEBUG_FLR_IMAGE_TIMESTAMP: 4d7db948
>
> MODULE_NAME: mydriver
>
> FAULTING_MODULE: 93d71000 mydriver
>
> DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
>
> PROCESS_NAME: System
>
> CURRENT_IRQL: 2
>
> LAST_CONTROL_TRANSFER: from 82b52f1f to 828fadfc
>
> STACK_TEXT:
> 87b27a90 82b52f1f 000000c4 00000062 8713cabc nt!KeBugCheckEx+0x1e
> 87b27ab0 82b57607 8713cabc 8713b700 93d71000
> nt!VerifierBugCheckIfAppropriate+0x30
> 87b27ac0 8282aeaa 8713ca60 8295dec8 8295dec8 nt!VfPoolCheckForLeaks+0x33
> 87b27afc 829af6b4 8713ca60 93d71000 40000000 nt!VfTargetDriversRemove+0x66
> 87b27b10 829af34d 829667e0 851eca70 00000000 nt!VfDriverUnloadImage+0x5e
> 87b27b48 829b05a2 8713ca60 ffffffff 00000000 nt!MiUnloadSystemImage+0x1c6
> 87b27b6c 82ad9a2b 8713ca60 851eeeb0 8714edf8 nt!MmUnloadSystemImage+0x36
> 87b27b84 82a3fa98 8714ee10 8714ee10 8714edf8 nt!IopDeleteDriver+0x38
> 87b27b9c 82887080 00000000 8245e800 8245e750 nt!ObpRemoveObjectRoutine+0x59
> 87b27bb0 82886ff0 8714ee10 829d7cd2 851eef78
> nt!ObfDereferenceObjectWithTag+0x88
> 87b27bb8 829d7cd2 851eef78 8245e738 8245e750 nt!ObfDereferenceObject+0xd
> 87b27bcc 82a3fa98 8245e750 8245e750 8245e738 nt!IopDeleteDevice+0x4e
> 87b27be4 82887080 00000000 82983e80 851e60d8 nt!ObpRemoveObjectRoutine+0x59
> 87b27bf8 82886ff0 8245e750 829d0384 a6cfacd0
> nt!ObfDereferenceObjectWithTag+0x88
> 87b27c00 829d0384 a6cfacd0 a69ed8f8 00000000 nt!ObfDereferenceObject+0xd
> 87b27c1c 829d130d a69ed8f8 a9e512d0 a9e512b0
> nt!PnpUnlinkDeviceRemovalRelations+0xe0
> 87b27cc4 829d3226 87b27cf4 00000000 a9e512b0
> nt!PnpProcessQueryRemoveAndEject+0xa05
> 87b27cdc 829d4d6e 00000000 90488c18 851eca70
> nt!PnpProcessTargetDeviceEvent+0x38
> 87b27d00 8288c03b 90488c18 00000000 851eca70 nt!PnpDeviceEventWorker+0x216
> 87b27d50 82a2c9df 00000001 8f61c205 00000000 nt!ExpWorkerThread+0x10d
> 87b27d90 828de1d9 8288bf2e 00000001 00000000 nt!PspSystemThreadStartup+0x9e
> 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19
>
>
> STACK_COMMAND: kb
>
> FOLLOWUP_NAME: MachineOwner
>
> FAILURE_BUCKET_ID: 0xc4_62_VRF_IMAGE_mydriver.sys
>
> BUCKET_ID: 0xc4_62_VRF_IMAGE_mydriver.sys
>
> Followup: MachineOwner
> ---------
>
> Then I type “!verifier 3 mydriver.sys”. It says the leaked pool is at paged
> pool, but it doesn’t supply the address of the leadked memory. Please look
> at below:
>
> 1: kd> !verifier 3 mydriver.sys
>
> Verify Level 9bb … enabled options are:
> Special pool
> Special irql
> All pool allocations checked on unload
> Io subsystem checking enabled
> Deadlock detection enabled
> DMA checking enabled
> Security checks enabled
> Miscellaneous checks enabled
>
> Summary of All Verifier Statistics
>
> RaiseIrqls 0x0
> AcquireSpinLocks 0x2032
> Synch Executions 0x0
> Trims 0x10
>
> Pool Allocations Attempted 0x53ec5
> Pool Allocations Succeeded 0x53ec5
> Pool Allocations Succeeded SpecialPool 0x53ec5
> Pool Allocations With NO TAG 0x0
> Pool Allocations Failed 0x0
> Resource Allocations Failed Deliberately 0x0
>
> Current paged pool allocations 0x1 for 00000030 bytes
> Peak paged pool allocations 0x7 for 00000600 bytes
> Current nonpaged pool allocations 0x0 for 00000000 bytes
> Peak nonpaged pool allocations 0x25 for 0000F744 bytes
>
> Driver Verification List
>
> Entry State NonPagedPool PagedPool Module
>
> ERROR: BaseAddress is NULL for suspect drivers list entry at 850a4b28.
>
> I don’t know what module the leaked pool is at.
>
> What should I do to find out the leakage function in my codes?
>
> Any help is appreciated!!!
>
> Joseph
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

–
Danny

3

Thanks for your help.
I would like to know how to check inside 850a4b28?
Is the below info helpful? Actually I could not find any from it.
Thanks again.

1: kd> u 850a4b28
850a4b28 c8df9582 enter 95DFh,82h
850a4b2c c8df9582 enter 95DFh,82h
850a4b30 0100 add dword ptr [eax],eax
850a4b32 0000 add byte ptr [eax],al
850a4b34 0000 add byte ptr [eax],al
850a4b36 0000 add byte ptr [eax],al
850a4b38 1400 adc al,0
850a4b3a 16 push ss

4

> Thanks for your help.

I would like to know how to check inside 850a4b28?
Is the below info helpful? Actually I could not find any from it.
Thanks again.

1: kd> u 850a4b28
850a4b28 c8df9582 enter 95DFh,82h
850a4b2c c8df9582 enter 95DFh,82h
850a4b30 0100 add dword ptr [eax],eax
850a4b32 0000 add byte ptr [eax],al
850a4b34 0000 add byte ptr [eax],al
850a4b36 0000 add byte ptr [eax],al
850a4b38 1400 adc al,0
850a4b3a 16 push ss

850a4b28 could be a LIST_ENTRY with FLink and BLink pointing to c8df9582
(eg a single item in the list). That’s just a guess though.

‘u’ is for disassembling code, and this isn’t code. Use ‘d’ for ‘display
memory’ instead, which should also show any ascii in there too.

James

5

Thank you James for your reply and advice.

1: kd> d 850a4b28
850a4b28 8295dfc8 8295dfc8 00000001 00000000
850a4b38 00160014 850a4b40 00700072 00680073
850a4b48 00690073 0073002e 00730079 00000000
850a4b58 04090007 54416656 850a4b60 00000000
850a4b68 8aa37ae8 000000ff 00000000 00000000
850a4b78 0000008a 00000008 00000000 00000007
850a4b88 828362bc 82836e62 8282ad3e 00000000
850a4b98 00000000 00000000 040c0009 70446d4d

It seems to me that we still can’t find any thing helpful. Could you please shed me any light on this?

Any comment would be greatly appreciated!

Joseph

6

>

Thank you James for your reply and advice.

1: kd> d 850a4b28
850a4b28 8295dfc8 8295dfc8 00000001 00000000
850a4b38 00160014 850a4b40 00700072 00680073
850a4b48 00690073 0073002e 00730079 00000000
850a4b58 04090007 54416656 850a4b60 00000000
850a4b68 8aa37ae8 000000ff 00000000 00000000
850a4b78 0000008a 00000008 00000000 00000007
850a4b88 828362bc 82836e62 8282ad3e 00000000
850a4b98 00000000 00000000 040c0009 70446d4d

It seems to me that we still can’t find any thing helpful. Could you
please
shed me any light on this?

Any comment would be greatly appreciated!

The first two 32 bit values may represent an element in a doubly linked
list. Try using ‘d’ on those, eg in this case:

d 8295dfc8

James

7

Thanks for your quick reply.

And the info around 8295dfc8 seem to pointer back to 850a4b28. And it still could not generage any ascii there.

1: kd> d 8295dfc8
8295dfc8 850a4b28 850a4b28 00000000 00000000
8295dfd8 00000000 00000000 00000001 00000000
8295dfe8 00000001 0000031b 0000031b 00000001
8295dff8 00000000 00000000 00000001 8713ca8c
8295e008 00000000 00000000 00000000 00000000
8295e018 00000000 00000000 00080002 00000001
8295e028 8295e028 8295e028 851ecf30 829667f0
8295e038 00000000 00000100 00001fff 00000000

1: kd> u 8295dfc8
nt!VfSuspectDriversList:
8295dfc8 284b0a sub byte ptr [ebx+0Ah],cl
8295dfcb 8528 test dword ptr [eax],ebp
8295dfcd 4b dec ebx
8295dfce 0a8500000000 or al,byte ptr [ebp]
8295dfd4 0000 add byte ptr [eax],al
8295dfd6 0000 add byte ptr [eax],al
8295dfd8 0000 add byte ptr [eax],al
8295dfda 0000 add byte ptr [eax],al

8

luming419@163.com wrote:

Thanks for your quick reply.

And the info around 8295dfc8 seem to pointer back to 850a4b28. And it still could not generage any ascii there.

1: kd> d 8295dfc8
8295dfc8 850a4b28 850a4b28 00000000 00000000
8295dfd8 00000000 00000000 00000001 00000000
8295dfe8 00000001 0000031b 0000031b 00000001
8295dff8 00000000 00000000 00000001 8713ca8c
8295e008 00000000 00000000 00000000 00000000
8295e018 00000000 00000000 00080002 00000001
8295e028 8295e028 8295e028 851ecf30 829667f0
8295e038 00000000 00000100 00001fff 00000000

1: kd> u 8295dfc8
nt!VfSuspectDriversList:
8295dfc8 284b0a sub byte ptr [ebx+0Ah],cl

Are you REALLY unable to extract anything useful from this? Do you see
the symbol that is attached to this piece of data? This is a linked
list inside Driver Verifier that points to its list of “suspect
drivers”. There is one driver on that list, at 850A4B28.

If I had to guess, I’d guess that 805A4B28 is probably a LIST_ENTRY in
the driver object of the suspect driver. But you already know which
driver is at fault, so this line of inquiry has reached a dead end.

Look, you already know from the dump the size of the block you are
leaking (one block, 48 bytes). That, by itself, should be enough for
you to find the bad allocation. If nothing else, create a wrapper
around ExAllocatePool that dumps the size of everything you allocate.
When you find a 48-byte allocation, that’s probably it (although there
could be more than one, of course).

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

9

I’ve used ExAllocatePoolWithTag() with poolmon specifying my tags to find leaks.

Larry C

10

If you’ve tagged each type of allocation you make with a unique tag,
finding the leak is pretty simple. Even if you use one tag for all of
your allocations, the size should be a huge clue.

Mark Roddy

On Fri, Mar 18, 2011 at 2:02 PM, wrote:
> I’ve used ExAllocatePoolWithTag() with poolmon specifying my tags to find leaks.
>
> Larry C
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

11

Yes, I have more than a handfull of tags that I used for my drivers. then I add my tag names to ‘pooltag.txt’ and let poolmon show me how the memory fluctuates during idle to high load to idle conditions. This points out the problem child quite nicely.

Larry C

12

Thank you all so much. Your comments made me reconsider the usage of Poolmon.

I made a mistake by using Poolmon.exe that I forgot the tag is usually specified in reverse order. So I didn’t find my tag. sad!

I’ve found my tags, but their allocations exactly match the bytes of memory freed. So it seems to be a resource leak.