Catching a memory leak in a WDF driver with the verifier, when the driver cannot be unloaded

NTDEV
1

I am dealing with a KMDF driver that I suspect has a memory leak in one of the WDF objects. I enabled the driver verifier for it, as well as the WDF verifier. The issue is that the driver in question does not seem to support unloading.

To test if the verifier can catch memory leaks when the OS is rebooting (and all the drivers are unloading) I wrote a small KMDF driver, introduced an intentional memory leak in it, enabled the verifier for that driver and tried to unload it manually. True to its form, the verifier then created a BSOD.

So I tried the same with my leaky driver but during the OS reboot. Unfortunately at that point, the verifier did not cause a BSOD.

Thus I’m wondering, if I need to do anything additional to enable verifier checks during the reboot?

Or, maybe there’s a way to force a driver to unload?

Any other suggestions.

2

Shutdown does not guarantee that your driver is unloaded, as you discovered. In particular, if you don’t support unload, shutdown is just going to ignore your driver.

The obvious answer is to support unload. If there is some compelling reason to not do that in your production version, then only support it in a test or debug version.

3

@Mark_Roddy unfortunately it’s not possible to support unloading. This is a boot critical driver that other drivers depend on, so there’s no way to unload it without crashing the OS.

I wonder if there’s another way to run verifier on it to test for memory leaks?

4

You could write your own “Allocator”. What I usually do is that I have my own functions for allocations that save all the allocations in a hash table with the location of the call. My code looks something like this:

#ifdef DEBUG_ALLOC
  #define InternalAllocateBytesWithTag(pool, bytes, tag, ...) ::detail::DebugAllocate(pool, bytes, (ULONG)tag, __VA_ARGS__)
  #define InternalAllocateBytes(pool, bytes, ...) InternalAllocateBytesWithTag(pool, bytes, 'Tagg', __VA_ARGS__)
#else
  #define InternalAllocateBytesWithTag(pool, bytes, tag) ::ExAllocatePoolWithTag(pool, bytes, (ULONG)tag)
  #define InternalAllocateBytes(pool, bytes, ...) InternalAllocateBytesWithTag(pool, bytes, 'Tagg')
#endif

  #define Allocate(type, pool) reinterpret_cast<type*>(InternalAllocateBytes(pool, sizeof(type), __FILE__, __LINE__))
  #define AllocateBytes(type, count, pool) reinterpret_cast<type*>(InternalAllocateBytes(pool, count, __FILE__, __LINE__))

InternalAllocateBytes then saves this in a hash table. On Free, you look up the address, if its not there, you can BugCheck because you are double freeiing, if successful, you just remove the entry from the table. At any point you can print all allocations and even check where they were made. I do this in a minifilter driver and I it doesnt seem to have any real performance impact. But if you do huge amounts of allocations/frees, you will start to notice it. You can also do some other cool memory checks, for example under and over allocating a bit and check for corruption when freeing, but you need to return correctly allocated memory.

Anyway what I would do in your case is find a point in time where the amount of allocated memory should be minimal, for example no drivers connected to it, and list all allocations. Or maybe print tags that you suspect are leaking? Or maybe do this only for the WDF object you suspect is leaking. But you need to ensure that what you match all custom allocations to their respective free functions, otherwise you will have false positives or the system hangs.

5

You could write your own memory ‘allocator’ that saves all the allocations in a hash table, the remove it on free. At any point you can print all allocations and determine what is leaking.

In my code I have something like this:

namespace detail {
  void* DebugAllocate(POOL_TYPE pool, SIZE_T bytes, ULONG tag, char const* file, ULONG line) {
  }

  void DebugFree(void* userPtr, ULONG tag) {
  }
}

#ifdef DEBUG_ALLOC
  #define InternalAllocateBytesWithTag(pool, bytes, tag, ...) ::detail::DebugAllocate(pool, bytes, (ULONG)tag, __VA_ARGS__)
  #define FreeWithTag(ptr, tag) ::detail::DebugFree(ptr, tag);
#else
  #define InternalAllocateBytesWithTag(pool, bytes, tag, ...) ::ExAllocatePoolWithTag(pool, bytes, (ULONG)tag)
  #define FreeWithTag(ptr, tag) ::ExFreePoolWithTag(ptr, tag)
#endif

  #define AllocateWithTag(type, pool, tag) reinterpret_cast<type*>(InternalAllocateBytesWithTag(pool, sizeof(type), tag, __FILE__, __LINE__))
  #define AllocateBytesWithTag(type, count, pool, tag) reinterpret_cast<type*>(InternalAllocateBytesWithTag(pool, count, tag, __FILE__, __LINE__))

In your case I would find a point in time where the amount of allocations is minimal and print them or make statistics out of them. Or you could use these functions for the part, where you suspect the memory leaks.

Edit: You dont have to do anything fancy in the allocation function, just call ExAllocatePoolWithTag, store it in a table and return it.

6

Or you can just give each instance in your code that allocates memory its own allocation tag. You can then track those allocations using the existing pool utilities.

7

It’s not my code, guys. I have access to the source code, but I am not the author. And this is a giant driver that cann’t be unloaded. Other drivers rely on it. Even building it takes some learning.

As for @sovak 's suggestion, I thought it too, but unfortunately there will be thousands and thousands entries in that list without any specifics other than the address and size.

It’s a mess, I know.

8

when creating wrappers for the alloc and free functions, the FILE FUNCTION and LINE macros make your job much easier. You can use them to create a macro that replaces the standard alloc / free call with your custom function and automatically provides context.

#define ExAllocatePool(PoolType, NumberOfBytes) (DebugAllocFunction_(PoolType, NumberOfBytes, FILE, FUNCTION, LINE))

just make sure this macro is not in scope where you implement DebugAllocFunction so it can call the real ExAllocatePool

9

@MBond2 the issue with your suggestion is that the allocation function may be called from some higher level function, which will give you a bunch of repeating function names and line numbers. Eg: WdfMemoryCreate, but there’s a bunch more.

10

Obviously, you have to select those functions where you suspect incorrect programming has been done. Knowing where to apply this is a big part of the job of the programmer.

If you want more information, you can collect stacks at the point of allocation / free. The allocate and free stacks generally should not align, but they could be useful for further diagnostics