I have a driver that bugchecks on unload in 8.1 with verifier pool tracking & special pool enabled because the driver (evidently) is not freeing some pool that it allocated.
The bugcheck is:
DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 00000062, A driver has forgotten to free its pool allocations prior to unloading.
Arg2: 8ab76c8c, name of the driver having the issue.
Arg3: 8ab80f70, verifier internal structure with driver information.
Arg4: 00000003, total # of (paged+nonpaged) allocations that weren’t freed.
Type !verifier 3 drivername.sys for info on the allocations
that were leaked that caused the bugcheck.
yet the output of !verifier 3 mydriver.sys does not show the allocations that have been leaked.
!verifier 3 mydriver.sys
… (output truncated to the interesting part) …
Driver Verification List
MODULE: 0x85027a18 mydriver.sys (Loaded)
Pool Allocation Statistics: ( NonPagedPool / PagedPool )
Current Pool Allocations: ( 0x00000003 / 0x00000000 )
Current Pool Bytes: ( 0x00000085 / 0x00000000 )
Peak Pool Allocations: ( 0x00000251 / 0x00000001 )
Peak Pool Bytes: ( 0x0000ce9f / 0x00000034 )
Contiguous Memory Bytes: 0x00000000
Peak Contiguous Memory Bytes: 0x00000000
Pool Allocations:
Address Length Tag Caller
Contiguous allocations are not displayed with public symbols.
Now in a previous run where I fixed the first four such leaks that list which was empty very nicely gave me the pool addresses of the leaked blocks and I was able to see what exactly they were.
Is there a way to go find out what allocations are left un-freed when !verifier 3 simply does not display them in that list?
Thanks,
Dave Cattley
Do you call AllocateCommonBuffer with sub-page length?
> Do you call AllocateCommonBuffer with sub-page length?
Nope.
I think this may be a problem with Verifier.
If I enable both Pool Tracking and Special Pool, I get the bugcheck. Verifier thinks I left precisely one allocation laying about of Size 0x27 bytes.
If I only enable Pool Tracking, Verifier thinks my allocations are properly balanced with de-allocations and does not bugcheck.
Since my driver does not change its behavior because Verifier shims allocations with the Special Pool, I can’t see how this could be my driver’s issue.
But how I go about proving that? Well, isn’t that what Verifier is for?
Thanks,
Dave Cattley
I found my problem (but not why Verifier behaves as it does).
I did actually have a leak that was being detected when special pool was enabled but not detected when special pool was disabled. I have no idea why.
Simply using !poolfind with the tags my driver uses eventually revealed (when special pool was disabled) the one leaked block and !verifier 80 did give me the stack back-trace to go fix it from.
But weird that when special pool is enabled, the counts are right but the tracking data is missing. And when special pool is disabled, the counts are wrong but the tracking data is correct.
Go figure.
Dave Cattley
From: xxxxx@msn.com
To: xxxxx@lists.osr.com
Subject: RE: [ntdev] !verifier analysis question
Date: Wed, 13 Nov 2013 15:14:50 -0500
> Do you call AllocateCommonBuffer with sub-page length?
Nope.
I think this may be a problem with Verifier.
If I enable both Pool Tracking and Special Pool, I get the bugcheck. Verifier thinks I left precisely one allocation laying about of Size 0x27 bytes.
If I only enable Pool Tracking, Verifier thinks my allocations are properly balanced with de-allocations and does not bugcheck.
Since my driver does not change its behavior because Verifier shims allocations with the Special Pool, I can’t see how this could be my driver’s issue.
But how I go about proving that? Well, isn’t that what Verifier is for?
Thanks,
Dave Cattley
Dave -
Win 8.1 will use the same bugcheck code and identifier for I/O Workitems that have been leaked (0xC4 / 0x62) as well (an nt!_IO_WORKITEM structure is around that size - 0x30 on x86), however I’m not positive it would be that. Otherwise, the mechanisms in place here are (AFAIK) unchanged through a few releases now.
I’d really appreciate your assistance in getting to the bottom of what Verifier (or the !verifier debugger extension) might be doing here. I’ve not seen an issue like this, and want to understand what the problem may be.
When you hit the bugcheck, can you paste the stack (or at least the function that called KeBugCheck)? Also - can you show the output of:
dt nt!_VF_TARGET_VERIFIED_DRIVER_DATA
Both should give me a better idea of at least where the check is happening and why, and I believe both will work with public symbols.
Thanks,
Michael
It looks like the debugger won’t show leaked full page allocations (“contiguous allocation”) without private symbols.
> It looks like the debugger won’t show leaked full page allocations (“contiguous allocation”) without private symbols.
So that is what a “contiguous allocation” means. Thanks for explaining that.
That could explain why disabling special pool allowed me to find the leaked object by tag and then look at the log/trace data to display the allocation stack backtrace.
Thanks,
Dave Cattley
Two different issues, as it turns out. I traded information with Dave off-forum and was able to confirm that both issues are in how information is presented and reported, and (thankfully) NOT an issue where Driver Verifier is improperly tracking information.
No crash when Special Pool not enabled:
The driver in question is an NDIS miniport driver; in Win8.1, Driver Verifier is now able to track the pool allocations and frees of the miniports themselves, and check for leaks when the driver is unloaded. In Win8 and previous, these allocations would show up as belonging to ndis.sys. This tracking for the miniports happens only when the Special Pool Flag is enabled; I don’t know (yet) if that was by design or if in error (and should be instead tied to the flag that would logically apply here: Pool Tracking).
When Special Pool flag is not enabled, the allocations made by a NDIS miniport driver show up as belonging to NDIS.sys (“!verifier 3 ndis.sys” output would show the allocations in that list).
Crash, but allocations not showing in list:
This is an issue with the Verifier debugger extension and how the pool alloc/free information is stored. The leaked allocations were all an odd number of bytes, and when walking through the list of allocations and frees for a driver, when the number of bytes is odd (non-even), that allocation is not displayed in the list.
The three allocations in the !analyze output from the first post add up to 0x85 bytes, meaning at least one had to be odd. As it turns out, all three were odd, and hence, none were displayed in the “!verifier 3 MyDriver.sys” output. (Had 2 been even and 1 odd, the 2 evens would’ve shown up, no matter their place in the list).
Thanks again for bringing these to our attention, Dave! Determining what to do and “fixing” these is already under way.
Michael,
Thanks for the detailed follow-up and explanations. I want to stress that
VRF did find my leak and now that I know to [go back to] looking for NDIS
owned allocations in the specific case you outline below as well as to have
my allocator logic ask for even byte counts, hunting for my next leak will
be even more effective.
Cheers,
Dave Cattley