Maybe a dumb question but say I want to make the memory allocations in the paged pool to minimize waste using the page size (say 4096). Allocating memory of 4096 on x64 results in verifier (debug build) showing 4112 bytes are allocated. I presume the extra 16 bytes have to do with tracking the memory? The returned buffer pointer is paged aligned. Does that mean an entire 4096-16 bytes are wasted do to the 16 extra bytes? If so, then allocating the buffersizealignedto4096 should have -16 added to it to prevent the waste or is it still wasted since the pointer is page aligned?
Don’t optimize for when verifier is on, it takes a large amount of overhead regardless of what you do. Optimize for the normal case AFTER you have measured your code and understand its behavior and impact to the system. Preemptive optimizations without data have been some of the worst code changes I have ever seen and didn’t actually make the system better as there was no measurement of their effectiveness
From: xxxxx@terabyteunlimited.com Sent: Tuesday, September 29, 2015 7:43 PM To: Windows System Software Devs Interest List Subject: [ntdev] Memory Allocation Question
Maybe a dumb question but say I want to make the memory allocations in the paged pool to minimize waste using the page size (say 4096). Allocating memory of 4096 on x64 results in verifier (debug build) showing 4112 bytes are allocated. I presume the extra 16 bytes have to do with tracking the memory? The returned buffer pointer is paged aligned. Does that mean an entire 4096-16 bytes are wasted do to the 16 extra bytes? If so, then allocating the buffersizealignedto4096 should have -16 added to it to prevent the waste or is it still wasted since the pointer is page aligned?
It’s all ready, just this last question of why it showing 4112 vs 4096 and wanting to ensure it doesn’t waste (4096-16) bytes. Or if my initial allocation should be 4096-16 for x64 ? or is that 16 bytes related to something else (say adding the tag to some chain)?
Well, there’s obviously overhead involved in tracking your allocation.
But, more to the point: If you have Special Pool enabled, Driver Verifier is allocating an extra page (or is it two) to ensure that you don’t overrun the page.
Don’t worry about how the pool allocator or Driver Verifier are written. Down that path lies madness.
It’s all ready, just this last question of why it showing 4112 vs 4096 and wanting to ensure it doesn’t waste (4096-16) bytes. Or if my initial allocation should be 4096-16 for x64 ? or is that 16 bytes related to something else (say adding the tag to some chain)?
You understand that there is ALWAYS overhead in a memory allocation
scheme, right? Someone, somewhere has to keep track of which chunks are
free, which chunks are allocated, and what size they are. The most
convenient way to do that is with a linked list. In addition, the
Windows kernel stores the “tag” word and some guard words so it can
detect heap corruption. The size of the overheadarea is an
implementation detail that is not documented and is entirely irrelevant.
Further, if you turn on the very useful “Special Pool” box in Verifier,
then EVERY allocation, no matter how small, occupies at least two pages.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Yes, I recall DOS used link list. Believe C libs do the same. Which is where the question came from, if I’m getting page aligned pointers and there was a linked list header of 16 bytes, it could be wasting 4096-16 bytes. But I doubt they would use that for dealing with pages, they probably use something like a bitmap, or another separate chain away from the buffer itself.
I’ll just not worry about that 16bytes and assume under the hood it is efficiently handling paged size memory allocations and that 16 bytes is somewhere else (verifier says 1 allocation of 4112 instead of 2 for 4112, but maybe it does something internally). The tag in windbg shows the same, 4112 bytes for the tag (but the tag is shared). It be cool if windbg would show allocation of the tag (or any internal mapping) separate from the buffer. Maybe it has to do with debug build too? But I’ll just have to trust (yikes) under the hood is doing what is expected…
If you’re curious, you can read a short article about how the pool manager does his allocation. This concentrates on shorter allocations, but at least it’s reasonably current: