> Using debug CRT actually makes pageheap less effective because
debug CRT adds its own padding between the end of the user
allocation and the start of the pageheap’s guard page.
Well, using page heap makes it less effective as well.
Hence the “#ifdef DEBUG” in my code, no thing of it gets into
the release version.
L.
> 1>
code in a bug :-
int main() {
void *ptr = 0;
int overrun = 1;
ptr = malloc(overrun);
while(overrun++)
{
if(!ptr)
while(1) Sleep(500);*((char*)ptr + (overrun+1)) = ‘a’;
printf(“\n%d\n”,overrun);
}return 0;
}2>
Enable FULL PAGE HEAP :-
gflags /p /enable test.exe /full3>
From project menu of visual studio 2010 made sure the build is “Release”
and “x64” (machine is x64)4> E:\installed\Debugging Tools for Windows (x64)>windbg -I
make windbg default debugger5> run the code as separate exe from cmd without debugger
output :-
2
3
4
5
6
7
8
9
10
11
12
13
14
after which windbg is seen catching the corruption. And I thought full
page heap is suppose to catch corruptions instantly.Any comments as to why full page heap sucks?
WINDBG is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
No, not at all. Allocations always take place on boundaries which are
multiples of the basic allocation unit, which is never less that 8.
Depending on various factors, the basic allocation quantum can be anywhere
between 24 and 72 bytes, and there is no guarantee that allocating from a
full-page heap will cause an instant error once the allocation goes beyond
the limit; it will not be detected until the allocation falls off the end
of the page, which is going to depend upon how much “slack space” there is
in the allocation. For example, if you ask for 1 byte, you will get AT
LEAST 8 bytes; if these are at the end of the page, then you can write
bytes 1…7 of this allocation without triggering an addressing error. But
if it is 16 bytes, then you can write bytes 1…15 without triggering an
error.
It is really, really important to understand how heaps work. There are
two essays on my Web site:
http://www.flounder.com/memory_allocation.htm
http://www.flounder.com/inside_storage_allocation.htm
I find that neither most would-be driver writers nor most application
programmers think of the heap as other than “magic”, and when you’ve got
memory problems, that is not going to be enough information to help. A
deeper understanding is required.
joe
> Gflags -p has /unaligned option which will detect small overruns
instantly. It will also break any code that assumes heap allocations are
8/16 bytes aligned.
Which becomes really important if there is any values that you are using
Interlocked…() operations with, which are assumed to be on the natural
alignment boundary for the operand size. Get them unaligned, and the
Interlocked…() operations are no longer guaranteed to work correctly.
joe
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: Friday, November 9, 2012 10:38 AM
To: Kernel Debugging Interest List
Subject: RE:[windbg] windows page heap not workingI guess malloc(1) reserves 16 bytes, giving me room to overrun.
This makes me doubt the usefulness of the tool.
It is basically saying “I will catch overruns iff they are big enough”
technically “overrunning the heap manager’s(or whatever) allocation size,
not user sizes”
WINDBG is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
WINDBG is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
I you run with the debug version of the allocator (in user space) those
bytes above the allocation (bytes 1…7, if the quantum is 8 bytes) are
filled with a pattern which the allocator calls “no man’s land”. If, upon
release of that block, the allegedly unused bytes are NOT this pattern you
will get a notification. Of course, this is perhaps billions of
instructions after the damage, but you WILL get a notification.
joe
I guess malloc(1) reserves 16 bytes, giving me room to overrun.
This makes me doubt the usefulness of the tool.
It is basically saying “I will catch overruns iff they are big enough”
technically “overrunning the heap manager’s(or whatever) allocation size,
not user sizes”
WINDBG is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
What I meant by “less effective” is that the padding added by debug CRT will interfere with pageheap’s ability to detect overruns instantly. So when you’re testing with pageheap it’s better to use retail CRT.
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Ladislav Zezula
Sent: Friday, November 9, 2012 12:53 PM
To: Kernel Debugging Interest List
Subject: Re[2]: [windbg] windows page heap not working
Using debug CRT actually makes pageheap less effective because debug
CRT adds its own padding between the end of the user allocation and
the start of the pageheap’s guard page.
Well, using page heap makes it less effective as well.
Hence the “#ifdef DEBUG” in my code, no thing of it gets into the release version.
> What I meant by “less effective” is that the padding added by debug CRT
will interfere with pageheap’s ability to detect overruns instantly. So
when you’re testing with pageheap it’s better to use retail CRT.
All allocators add padding; the storage is a multiple of 8 for maximum
compatibiilty with natural alignment boundaries, meaning any performance
or correctness problems caused by unaligned data remain the problem of the
programmer. The heap will guarantee consistent alignment. The retail CRT
does this also; I have written programs that demonstrate this.
joe
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ladislav Zezula
Sent: Friday, November 9, 2012 12:53 PM
To: Kernel Debugging Interest List
Subject: Re[2]: [windbg] windows page heap not working> Using debug CRT actually makes pageheap less effective because debug
> CRT adds its own padding between the end of the user allocation and
> the start of the pageheap’s guard page.Well, using page heap makes it less effective as well.
Hence the “#ifdef DEBUG” in my code, no thing of it gets into the release
version.
WINDBG is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
On 09-Nov-2012 21:39, Pavel Lebedynskiy wrote:
Gflags -p has /unaligned option which will detect small overruns instantly. It will also break any code that assumes heap allocations are 8/16 bytes aligned.
Pool allocations can be not aligned??
Then, what is the meaning of MEMORY_ALLOCATION_ALIGNMENT defined in ntdef.h?
– pa
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Friday, November 9, 2012 10:38 AM
To: Kernel Debugging Interest List
Subject: RE:[windbg] windows page heap not workingI guess malloc(1) reserves 16 bytes, giving me room to overrun.
This makes me doubt the usefulness of the tool.
It is basically saying “I will catch overruns iff they are big enough”
technically “overrunning the heap manager’s(or whatever) allocation size, not user sizes”
Yes, and your point is…?
Can you point to a place in the documentation where pageheap is specified
as having this guarantee? If you find it, it should be reported as a
documentation error.
What I recommend for app programming is for GUI apps to embed in the
message loop
ASSERT(_heapchk() == HEAP_OK);
[disclaimer: I am doing this from memory; check the docs for the correct
names, and if they differ from these, consider this pseudo-code]
That way, if there’s any damage, you have a much better idea which path is
responsible. Then, along that path, sprinkle more of these in strategic
locations.
If you don’t have a GUI app, the generalization of this technique is left
as an Exercise For The Reader.
And why would you think that malloc(1) allocates only one byte?
An all-too-common question in the MFC newsgroup was of the form:
“I’m doing a malloc(1) and according to my calculations I should be able
to do this 1,000,000,000 times without problem. But it fails after NN,NNN
allocations. I have mGB” (usually 4 or 8) “of physical memory. This
should not fail” (How many fundamental errors of understanding can you put
in one question?)
In the debug version of the VS 6 release, malloc(1) allocated 72 bytes of
memory, counting the internal storage headers. Nothing I have ever seen
requires an allocator to do other than meet the request given, and there
is no reason to presume you will get exactly that, or even the value you
asked for rounded up to the next N-byte boundary, for some value of N
being 8 or 16. For example, to minimize fragmentation, the VS 6 release
library would round the string size up to 64, 128, or 256. To avoid
memory fragmentation, I once worked with an allocator that would refuse to
split a block if the resulting free block was smaller than one-sigma from
the mean allocation. It did floating-point computations on a 1968
mainframe because if it saved one page fault it would pay for the slow FPU
on that mainframe.
The idea that gflags could override the alignment is, frankly, dowright
scary. It could break every Interlocked…() in the kernel!
joe
I guess malloc(1) reserves 16 bytes, giving me room to overrun.
This makes me doubt the usefulness of the tool.
It is basically saying “I will catch overruns iff they are big enough”
technically “overrunning the heap manager’s(or whatever) allocation size,
not user sizes”
WINDBG is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Pool allocations are aligned to at least that constant.
From: Pavel A
Sent: 11/11/2012 5:53
To: Kernel Debugging Interest List
Subject: Re:[windbg] windows page heap not working
On 09-Nov-2012 21:39, Pavel Lebedynskiy wrote:
Gflags -p has /unaligned option which will detect small overruns instantly. It will also break any code that assumes heap allocations are 8/16 bytes aligned.
Pool allocations can be not aligned??
Then, what is the meaning of MEMORY_ALLOCATION_ALIGNMENT defined in ntdef.h?
– pa
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Friday, November 9, 2012 10:38 AM
To: Kernel Debugging Interest List
Subject: RE:[windbg] windows page heap not workingI guess malloc(1) reserves 16 bytes, giving me room to overrun.
This makes me doubt the usefulness of the tool.
It is basically saying “I will catch overruns iff they are big enough”
technically “overrunning the heap manager’s(or whatever) allocation size, not user sizes”
WINDBG is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
1>
Nobody here is saying malloc(1) allocates 1 byte.
The question is : since documentation says that full page heap will catch
corruption as it happens, is it completely valid to ask if corruption here
means over-running even 1 byte of user allocation or over-running the
allocation boundary after the user allocation.
2>
Regarding the interlocked operations, I will always ensure that
any variable/mem passed as parameter to any interlocked is always
aligned explicitly. Thus the “/unaligned” attribute mentioned by “Pavel A” is
useful to me and not as scary as you find it to be.
-Gaurav
On Sun, Nov 11, 2012 at 10:48 PM, wrote:
> Yes, and your point is…?
>
> Can you point to a place in the documentation where pageheap is specified
> as having this guarantee? If you find it, it should be reported as a
> documentation error.
>
> What I recommend for app programming is for GUI apps to embed in the
> message loop
> ASSERT(_heapchk() == HEAP_OK);
> [disclaimer: I am doing this from memory; check the docs for the correct
> names, and if they differ from these, consider this pseudo-code]
>
> That way, if there’s any damage, you have a much better idea which path is
> responsible. Then, along that path, sprinkle more of these in strategic
> locations.
>
> If you don’t have a GUI app, the generalization of this technique is left
> as an Exercise For The Reader.
>
> And why would you think that malloc(1) allocates only one byte?
>
> An all-too-common question in the MFC newsgroup was of the form:
>
> “I’m doing a malloc(1) and according to my calculations I should be able
> to do this 1,000,000,000 times without problem. But it fails after NN,NNN
> allocations. I have mGB” (usually 4 or 8) “of physical memory. This
> should not fail” (How many fundamental errors of understanding can you put
> in one question?)
>
> In the debug version of the VS 6 release, malloc(1) allocated 72 bytes of
> memory, counting the internal storage headers. Nothing I have ever seen
> requires an allocator to do other than meet the request given, and there
> is no reason to presume you will get exactly that, or even the value you
> asked for rounded up to the next N-byte boundary, for some value of N
> being 8 or 16. For example, to minimize fragmentation, the VS 6 release
> library would round the string size up to 64, 128, or 256. To avoid
> memory fragmentation, I once worked with an allocator that would refuse to
> split a block if the resulting free block was smaller than one-sigma from
> the mean allocation. It did floating-point computations on a 1968
> mainframe because if it saved one page fault it would pay for the slow FPU
> on that mainframe.
>
> The idea that gflags could override the alignment is, frankly, dowright
> scary. It could break every Interlocked…() in the kernel!
> joe
> > I guess malloc(1) reserves 16 bytes, giving me room to overrun.
> >
> > This makes me doubt the usefulness of the tool.
> > It is basically saying “I will catch overruns iff they are big enough”
> > technically “overrunning the heap manager’s(or whatever) allocation size,
> > not user sizes”
> >
> > —
> > WINDBG is sponsored by OSR
> >
> > For our schedule of WDF, WDM, debugging and other seminars visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
> >
>
>
>
> —
> WINDBG is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> 1>
Nobody here is saying malloc(1) allocates 1 byte.
The question is : since documentation says that full page heap will catch
corruption as it happens, is it completely valid to ask if corruption here
means over-running even 1 byte of user allocation or over-running the
allocation boundary after the user allocation.
Can you give a reference to that page? It needs to be more explicit about
what it says, since if what it says is what you are stating here, it must
be wrong. It should not make that statement.
2>
Regarding the interlocked operations, I will always ensure that
any variable/mem passed as parameter to any interlocked is always
aligned explicitly. Thus the “/unaligned” attribute mentioned by “Pavel A”
is
useful to me and not as scary as you find it to be.
If the allocator is allowed to misalign allocations, how can you “ensure”
that a value is properly aligned? You have no control over it! Consider
typedef struct {
long count;
unsigned char data[5];
} whatever, *pwhatever;
This has to be aligned so that the last byte is at address 4095 bytes into
the page, which puts ‘&data[0]’ at offset 4091 and therefore ‘&count’ at
offset 4097. Be afraid. Be very afraid.
joe
-Gaurav
On Sun, Nov 11, 2012 at 10:48 PM, wrote:
>
>> Yes, and your point is…?
>>
>> Can you point to a place in the documentation where pageheap is
>> specified
>> as having this guarantee? If you find it, it should be reported as a
>> documentation error.
>>
>> What I recommend for app programming is for GUI apps to embed in the
>> message loop
>> ASSERT(_heapchk() == HEAP_OK);
>> [disclaimer: I am doing this from memory; check the docs for the correct
>> names, and if they differ from these, consider this pseudo-code]
>>
>> That way, if there’s any damage, you have a much better idea which path
>> is
>> responsible. Then, along that path, sprinkle more of these in strategic
>> locations.
>>
>> If you don’t have a GUI app, the generalization of this technique is
>> left
>> as an Exercise For The Reader.
>>
>> And why would you think that malloc(1) allocates only one byte?
>>
>> An all-too-common question in the MFC newsgroup was of the form:
>>
>> “I’m doing a malloc(1) and according to my calculations I should be able
>> to do this 1,000,000,000 times without problem. But it fails after
>> NN,NNN
>> allocations. I have mGB” (usually 4 or 8) “of physical memory. This
>> should not fail” (How many fundamental errors of understanding can you
>> put
>> in one question?)
>>
>> In the debug version of the VS 6 release, malloc(1) allocated 72 bytes
>> of
>> memory, counting the internal storage headers. Nothing I have ever seen
>> requires an allocator to do other than meet the request given, and there
>> is no reason to presume you will get exactly that, or even the value you
>> asked for rounded up to the next N-byte boundary, for some value of N
>> being 8 or 16. For example, to minimize fragmentation, the VS 6 release
>> library would round the string size up to 64, 128, or 256. To avoid
>> memory fragmentation, I once worked with an allocator that would refuse
>> to
>> split a block if the resulting free block was smaller than one-sigma
>> from
>> the mean allocation. It did floating-point computations on a 1968
>> mainframe because if it saved one page fault it would pay for the slow
>> FPU
>> on that mainframe.
>>
>> The idea that gflags could override the alignment is, frankly, dowright
>> scary. It could break every Interlocked…() in the kernel!
>> joe
>> > I guess malloc(1) reserves 16 bytes, giving me room to overrun.
>> >
>> > This makes me doubt the usefulness of the tool.
>> > It is basically saying “I will catch overruns iff they are big enough”
>> > technically “overrunning the heap manager’s(or whatever) allocation
>> size,
>> > not user sizes”
>> >
>> > —
>> > WINDBG is sponsored by OSR
>> >
>> > For our schedule of WDF, WDM, debugging and other seminars visit:
>> > http://www.osr.com/seminars
>> >
>> > To unsubscribe, visit the List Server section of OSR Online at
>> > http://www.osronline.com/page.cfm?name=ListServer
>> >
>>
>>
>>
>> —
>> WINDBG is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
> —
> WINDBG is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
1>
See the table :-
http://support.microsoft.com/kb/286470?wa=wsignin1.0
Access after end of block Caught upon free Caught instantly
here the definition of “block” is debatable and hence not clear.
2>
A combination of align directive and/or aligned malloc can be used to
ensure the alignment explicitly.
-Gaurav
On Mon, Nov 12, 2012 at 4:06 PM, wrote:
> > 1>
> > Nobody here is saying malloc(1) allocates 1 byte.
> >
> > The question is : since documentation says that full page heap will catch
> > corruption as it happens, is it completely valid to ask if corruption
> here
> > means over-running even 1 byte of user allocation or over-running the
> > allocation boundary after the user allocation.
>
> Can you give a reference to that page? It needs to be more explicit about
> what it says, since if what it says is what you are stating here, it must
> be wrong. It should not make that statement.
>
> >
> > 2>
> > Regarding the interlocked operations, I will always ensure that
> > any variable/mem passed as parameter to any interlocked is always
> > aligned explicitly. Thus the “/unaligned” attribute mentioned by “Pavel
> A”
> > is
> > useful to me and not as scary as you find it to be.
> >
> If the allocator is allowed to misalign allocations, how can you “ensure”
> that a value is properly aligned? You have no control over it! Consider
>
> typedef struct {
> long count;
> unsigned char data[5];
> } whatever, *pwhatever;
>
>
> This has to be aligned so that the last byte is at address 4095 bytes into
> the page, which puts ‘&data[0]’ at offset 4091 and therefore ‘&count’ at
> offset 4097. Be afraid. Be very afraid.
> joe
>
> > -Gaurav
> >
> > On Sun, Nov 11, 2012 at 10:48 PM, wrote:
> >
> >> Yes, and your point is…?
> >>
> >> Can you point to a place in the documentation where pageheap is
> >> specified
> >> as having this guarantee? If you find it, it should be reported as a
> >> documentation error.
> >>
> >> What I recommend for app programming is for GUI apps to embed in the
> >> message loop
> >> ASSERT(_heapchk() == HEAP_OK);
> >> [disclaimer: I am doing this from memory; check the docs for the correct
> >> names, and if they differ from these, consider this pseudo-code]
> >>
> >> That way, if there’s any damage, you have a much better idea which path
> >> is
> >> responsible. Then, along that path, sprinkle more of these in strategic
> >> locations.
> >>
> >> If you don’t have a GUI app, the generalization of this technique is
> >> left
> >> as an Exercise For The Reader.
> >>
> >> And why would you think that malloc(1) allocates only one byte?
> >>
> >> An all-too-common question in the MFC newsgroup was of the form:
> >>
> >> “I’m doing a malloc(1) and according to my calculations I should be able
> >> to do this 1,000,000,000 times without problem. But it fails after
> >> NN,NNN
> >> allocations. I have mGB” (usually 4 or 8) “of physical memory. This
> >> should not fail” (How many fundamental errors of understanding can you
> >> put
> >> in one question?)
> >>
> >> In the debug version of the VS 6 release, malloc(1) allocated 72 bytes
> >> of
> >> memory, counting the internal storage headers. Nothing I have ever seen
> >> requires an allocator to do other than meet the request given, and there
> >> is no reason to presume you will get exactly that, or even the value you
> >> asked for rounded up to the next N-byte boundary, for some value of N
> >> being 8 or 16. For example, to minimize fragmentation, the VS 6 release
> >> library would round the string size up to 64, 128, or 256. To avoid
> >> memory fragmentation, I once worked with an allocator that would refuse
> >> to
> >> split a block if the resulting free block was smaller than one-sigma
> >> from
> >> the mean allocation. It did floating-point computations on a 1968
> >> mainframe because if it saved one page fault it would pay for the slow
> >> FPU
> >> on that mainframe.
> >>
> >> The idea that gflags could override the alignment is, frankly, dowright
> >> scary. It could break every Interlocked…() in the kernel!
> >> joe
> >> > I guess malloc(1) reserves 16 bytes, giving me room to overrun.
> >> >
> >> > This makes me doubt the usefulness of the tool.
> >> > It is basically saying “I will catch overruns iff they are big enough”
> >> > technically “overrunning the heap manager’s(or whatever) allocation
> >> size,
> >> > not user sizes”
> >> >
> >> > —
> >> > WINDBG is sponsored by OSR
> >> >
> >> > For our schedule of WDF, WDM, debugging and other seminars visit:
> >> > http://www.osr.com/seminars
> >> >
> >> > To unsubscribe, visit the List Server section of OSR Online at
> >> > http://www.osronline.com/page.cfm?name=ListServer
> >> >
> >>
> >>
> >>
> >> —
> >> WINDBG is sponsored by OSR
> >>
> >> For our schedule of WDF, WDM, debugging and other seminars visit:
> >> http://www.osr.com/seminars
> >>
> >> To unsubscribe, visit the List Server section of OSR Online at
> >> http://www.osronline.com/page.cfm?name=ListServer
> >>
> >
> > —
> > WINDBG is sponsored by OSR
> >
> > For our schedule of WDF, WDM, debugging and other seminars visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
>
>
>
> —
> WINDBG is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>