AppVerifier complains about "freeing memory with Critical Section", but I am sure I did DeleteCritic

I suspect AppVerifier is wrong here, but I am not sure and would like
a second opinion, from someone on this lists who knows these parts of
AppVerifier more closely.

OS is Windows 5.02 x64 uniprocessor workstation (aka XP x64) SP2
WinDbg is version 6.7.0005.1

The scenario: A 64 bit service dynamically loads some "JIT-ed" code
into memory and runs it. After running for some time, the JIT-ed code
frees its resources and then the memory containing it is freed. Within
the allocated memory for the JIT-ed code/data are some
RTL_CRITICAL_SECTION objects initialized with
kernel32!InitializeCriticalSection and later deinitialized with
kernel32!DeleteCriticalSection . I have enabled AppVerifier for the
process with gflags.exe, and later attach to the service with WinDbg to
see and debug any AppVerifier induced messages.

When the memory block containing the JIT-ed code and its data is freed,
AppVerifier complains like this:

===========================================================
VERIFIER STOP 0000000000000212: pid 0x4BC: releasing virtual memory
containing active critical section

0000000003CF9748 : Critical section address
0000000000000000 : Initialization stack trace. Use dds to dump it if
non-NULL.
0000000003CE0000 : Memory block address
0000000000022000 : Memory block size

This verifier stop is continuable.
After debugging it use `go' to continue.

But I am almost sure I already deinitialized that critical section, in
particular:

!locks -v does not list 3CE0000 as a known critical section

A dump of address 3CE0000 shows that at least the first 16 bytes are 0

Before the AppVerifier complaint I set the following 2 breakpoints
bp ntdll!RtlDeleteCriticalSection ".echo DELCRIT;? @ecx;G"
bp ntdll!RtlInitializeCriticalSectionAndSpinCount ".echo INICRIT;?
@ecx;G"

Which resulted in the following output
... Lots of other critical sections reported as INICRIT/DELCRIT ...
INICRIT
Evaluate expression: 63936328 = 0000000003cf9748 ... Lots of other critical sections reported as INICRIT/DELCRIT ... DELCRIT Evaluate expression: 63936328 = 0000000003cf9748

And then the AppVerifier complaint above.

--
Jakob B?hm, M.Sc.Eng. * xxxxx@netop.com * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right.
Information in this posting may not be the official position of Danware
Data A/S, only the personal opinions of the author.

Is it possible that the CRITICAL_SECTION has not being set by the LeaveCriticalSection?

----- Original Message ----
From: Jakob Bohm
To: Kernel Debugging Interest List
Sent: Wednesday, October 1, 2008 11:24:55 AM
Subject: [windbg] AppVerifier complains about “freeing memory with Critical Section”, but I am sure I did DeleteCriticalSection()

I suspect AppVerifier is wrong here, but I am not sure and would like
a second opinion, from someone on this lists who knows these parts of
AppVerifier more closely.

OS is Windows 5.02 x64 uniprocessor workstation (aka XP x64) SP2
WinDbg is version 6.7.0005.1

The scenario: A 64 bit service dynamically loads some “JIT-ed” code
into memory and runs it. After running for some time, the JIT-ed code
frees its resources and then the memory containing it is freed. Within
the allocated memory for the JIT-ed code/data are some
RTL_CRITICAL_SECTION objects initialized with
kernel32!InitializeCriticalSection and later deinitialized with
kernel32!DeleteCriticalSection . I have enabled AppVerifier for the
process with gflags.exe, and later attach to the service with WinDbg to
see and debug any AppVerifier induced messages.

When the memory block containing the JIT-ed code and its data is freed,
AppVerifier complains like this:

===========================================================
VERIFIER STOP 0000000000000212: pid 0x4BC: releasing virtual memory
containing active critical section

0000000003CF9748 : Critical section address
0000000000000000 : Initialization stack trace. Use dds to dump it if
non-NULL.
0000000003CE0000 : Memory block address
0000000000022000 : Memory block size
===========================================================
This verifier stop is continuable.
After debugging it use go' to continue.<br>===<br><br>But I am almost sure I already deinitialized that critical section, in<br>particular:<br><br>!locks -v does not list 3CE0000 as a known critical section<br><br>A dump of address 3CE0000 shows that at least the first 16 bytes are 0<br><br>Before the AppVerifier complaint I set the following 2 breakpoints<br> bp ntdll!RtlDeleteCriticalSection ".echo DELCRIT;? @ecx;G"<br> bp ntdll!RtlInitializeCriticalSectionAndSpinCount ".echo INICRIT;? <br>@ecx;G"<br><br>Which resulted in the following output<br>... Lots of other critical sections reported as INICRIT/DELCRIT ...<br>INICRIT<br>Evaluate expression: 63936328 = 0000000003cf9748
… Lots of other critical sections reported as INICRIT/DELCRIT …
DELCRIT
Evaluate expression: 63936328 = 00000000`03cf9748

And then the AppVerifier complaint above.


Jakob B?hm, M.Sc.Eng. * xxxxx@netop.com * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right.
Information in this posting may not be the official position of Danware
Data A/S, only the personal opinions of the author.


You are currently subscribed to windbg as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

I just tested that, and no, the CRITICAL_SECTION objects were not
busy when deleted.

I changed the first breakpoint to

bp ntdll!RtlDeleteCriticalSection “.echo DELCRIT;!critsec @rcx;G”
And the output showed that each critical section was completely free
when it was deleted with DeleteCriticalSection.

When AppVerifier stopped the process microseconds later, the
CRITICAL_SECTION (all 0x30 bytes of it) were now all 0, which is
not valid for an initialized CRITICAL_SECTION, but perfectly
valid for a piece of uninitialized memory about to be freed.

Anyone with another idea?

Calin Iaru wrote:

Is it possible that the CRITICAL_SECTION has not being set by the LeaveCriticalSection?

> From: Jakob Bohm
> > To: Kernel Debugging Interest List
> > Sent: Wednesday, October 1, 2008 11:24:55 AM
> > Subject: [windbg] AppVerifier complains about “freeing memory with Critical Section”, but I am sure I did DeleteCriticalSection()
> >
> > I suspect AppVerifier is wrong here, but I am not sure and would like
> > a second opinion, from someone on this lists who knows these parts of
> > AppVerifier more closely.
> >
> > OS is Windows 5.02 x64 uniprocessor workstation (aka XP x64) SP2
> > WinDbg is version 6.7.0005.1
> >
> > The scenario: A 64 bit service dynamically loads some “JIT-ed” code
> > into memory and runs it. After running for some time, the JIT-ed code
> > frees its resources and then the memory containing it is freed. Within
> > the allocated memory for the JIT-ed code/data are some
> > RTL_CRITICAL_SECTION objects initialized with
> > kernel32!InitializeCriticalSection and later deinitialized with
> > kernel32!DeleteCriticalSection . I have enabled AppVerifier for the
> > process with gflags.exe, and later attach to the service with WinDbg to
> > see and debug any AppVerifier induced messages.
> >
> > When the memory block containing the JIT-ed code and its data is freed,
> > AppVerifier complains like this:
> >
> > ===========================================================
> > VERIFIER STOP 0000000000000212: pid 0x4BC: releasing virtual memory
> > containing active critical section
> >
> > 0000000003CF9748 : Critical section address
> > 0000000000000000 : Initialization stack trace. Use dds to dump it if
> > non-NULL.
> > 0000000003CE0000 : Memory block address
> > 0000000000022000 : Memory block size
> > ===========================================================
> > This verifier stop is continuable.
> > After debugging it use go' to continue.<br>&gt; &gt; ===<br>&gt; &gt; <br>&gt; &gt; But I am almost sure I already deinitialized that critical section, in<br>&gt; &gt; particular:<br>&gt; &gt; <br>&gt; &gt; !locks -v does not list 3CE0000 as a known critical section<br>&gt; &gt; <br>&gt; &gt; A dump of address 3CE0000 shows that at least the first 16 bytes are 0<br>&gt; &gt; <br>&gt; &gt; Before the AppVerifier complaint I set the following 2 breakpoints<br>&gt; &gt; bp ntdll!RtlDeleteCriticalSection ".echo DELCRIT;? @ecx;G"<br>&gt; &gt; bp ntdll!RtlInitializeCriticalSectionAndSpinCount ".echo INICRIT;? <br>&gt; &gt; @ecx;G"<br>&gt; &gt; <br>&gt; &gt; Which resulted in the following output<br>&gt; &gt; ... Lots of other critical sections reported as INICRIT/DELCRIT ...<br>&gt; &gt; INICRIT<br>&gt; &gt; Evaluate expression: 63936328 = 0000000003cf9748
> > … Lots of other critical sections reported as INICRIT/DELCRIT …
> > DELCRIT
> > Evaluate expression: 63936328 = 00000000`03cf9748
> >
> > And then the AppVerifier complaint above.
> >
> >
> >


Jakob B?hm, M.Sc.Eng. * xxxxx@danware.dk * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right.
Information in this posting may not be the official position of
Danware Data A/S, only the personal opinions of the author.

How are you retrieving the addresses of InitializeCriticalSection/LeaveCriticalSection?

IIRC, avrf does a dllimport hook to arrange to be called when kernel32!InitializeCriticalSection/kernel32!DeleteCriticalSection, and these, among other things, insert/remove the crit sec into a tracking list that is scanned when memory is decommitted.

Now, if you manage to directly call RtlDeleteCriticalSection in the delete path without going through the avrf hook, you may be able to trick avrf into thinking that you’ve got a leak when you do not really.

Simple static dllimports and GetProcAddress should not result in this behavior, however, if you are doing something more exotic, then you might run into a situation similar to what you described.

  • S

-----Original Message-----
From: Jakob Bohm
Sent: Wednesday, October 01, 2008 04:26
To: Kernel Debugging Interest List
Subject: [windbg] AppVerifier complains about “freeing memory with Critical Section”, but I am sure I did DeleteCriticalSection()

I suspect AppVerifier is wrong here, but I am not sure and would like
a second opinion, from someone on this lists who knows these parts of
AppVerifier more closely.

OS is Windows 5.02 x64 uniprocessor workstation (aka XP x64) SP2
WinDbg is version 6.7.0005.1

The scenario: A 64 bit service dynamically loads some “JIT-ed” code
into memory and runs it. After running for some time, the JIT-ed code
frees its resources and then the memory containing it is freed. Within
the allocated memory for the JIT-ed code/data are some
RTL_CRITICAL_SECTION objects initialized with
kernel32!InitializeCriticalSection and later deinitialized with
kernel32!DeleteCriticalSection . I have enabled AppVerifier for the
process with gflags.exe, and later attach to the service with WinDbg to
see and debug any AppVerifier induced messages.

When the memory block containing the JIT-ed code and its data is freed,
AppVerifier complains like this:

===========================================================
VERIFIER STOP 0000000000000212: pid 0x4BC: releasing virtual memory
containing active critical section

0000000003CF9748 : Critical section address
0000000000000000 : Initialization stack trace. Use dds to dump it if
non-NULL.
0000000003CE0000 : Memory block address
0000000000022000 : Memory block size
===========================================================
This verifier stop is continuable.
After debugging it use go' to continue.<br>===<br><br>But I am almost sure I already deinitialized that critical section, in<br>particular:<br><br>!locks -v does not list 3CE0000 as a known critical section<br><br>A dump of address 3CE0000 shows that at least the first 16 bytes are 0<br><br>Before the AppVerifier complaint I set the following 2 breakpoints<br> bp ntdll!RtlDeleteCriticalSection ".echo DELCRIT;? @ecx;G"<br> bp ntdll!RtlInitializeCriticalSectionAndSpinCount ".echo INICRIT;?<br>@ecx;G"<br><br>Which resulted in the following output<br>... Lots of other critical sections reported as INICRIT/DELCRIT ...<br>INICRIT<br>Evaluate expression: 63936328 = 0000000003cf9748
… Lots of other critical sections reported as INICRIT/DELCRIT …
DELCRIT
Evaluate expression: 63936328 = 00000000`03cf9748

And then the AppVerifier complaint above.


Jakob B?hm, M.Sc.Eng. * xxxxx@netop.com * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right.
Information in this posting may not be the official position of Danware
Data A/S, only the personal opinions of the author.


You are currently subscribed to windbg as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Well, I am doing something more exotic, but strangely I do the same
for InitializeCriticalSection, in which case I wonder how avrf catches
one, but not the other.

One possibility though: If avrf catches imports of the ntdll
entrypoints, then InitializeCriticalSection() goes through a
front in another system DLL which does a normal static import of
ntdll!RtlInitializeCriticalSectionAndSpinCount(), while
DeleteCriticalSection() is a simple forwarder import causing my
code to call ntdll!RtlDeleteCriticalSection() directly.

Worth looking into though.

Skywing wrote:

How are you retrieving the addresses of InitializeCriticalSection/LeaveCriticalSection?

IIRC, avrf does a dllimport hook to arrange to be called when
kernel32!InitializeCriticalSection/kernel32!DeleteCriticalSection, and
these, among other things, insert/remove the crit sec into a tracking
list that is scanned when memory is decommitted.

Now, if you manage to directly call RtlDeleteCriticalSection in the
delete path without going through the avrf hook, you may be able to
trick avrf into thinking that you’ve got a leak when you do not really.

Simple static dllimports and GetProcAddress should not result in this
behavior, however, if you are doing something more exotic, then you
might run into a situation similar to what you described.

  • S

> From: Jakob Bohm
>> Sent: Wednesday, October 01, 2008 04:26
>> To: Kernel Debugging Interest List
>> Subject: [windbg] AppVerifier complains about “freeing memory with Critical Section”, but I am sure I did DeleteCriticalSection()
>>
>>
>> I suspect AppVerifier is wrong here, but I am not sure and would like
>> a second opinion, from someone on this lists who knows these parts of
>> AppVerifier more closely.
>>
>> OS is Windows 5.02 x64 uniprocessor workstation (aka XP x64) SP2
>> WinDbg is version 6.7.0005.1
>>
>> The scenario: A 64 bit service dynamically loads some “JIT-ed” code
>> into memory and runs it. After running for some time, the JIT-ed code
>> frees its resources and then the memory containing it is freed. Within
>> the allocated memory for the JIT-ed code/data are some
>> RTL_CRITICAL_SECTION objects initialized with
>> kernel32!InitializeCriticalSection and later deinitialized with
>> kernel32!DeleteCriticalSection . I have enabled AppVerifier for the
>> process with gflags.exe, and later attach to the service with WinDbg to
>> see and debug any AppVerifier induced messages.
>>
>> When the memory block containing the JIT-ed code and its data is freed,
>> AppVerifier complains like this:
>>
>> ===========================================================
>> VERIFIER STOP 0000000000000212: pid 0x4BC: releasing virtual memory
>> containing active critical section
>>
>> 0000000003CF9748 : Critical section address
>> 0000000000000000 : Initialization stack trace. Use dds to dump it if
>> non-NULL.
>> 0000000003CE0000 : Memory block address
>> 0000000000022000 : Memory block size
>> ===========================================================
>> This verifier stop is continuable.
>> After debugging it use go' to continue.<br>&gt;&gt; ===<br>&gt;&gt; <br>&gt;&gt; But I am almost sure I already deinitialized that critical section, in<br>&gt;&gt; particular:<br>&gt;&gt; <br>&gt;&gt; !locks -v does not list 3CE0000 as a known critical section<br>&gt;&gt; <br>&gt;&gt; A dump of address 3CE0000 shows that at least the first 16 bytes are 0<br>&gt;&gt; <br>&gt;&gt; Before the AppVerifier complaint I set the following 2 breakpoints<br>&gt;&gt; bp ntdll!RtlDeleteCriticalSection ".echo DELCRIT;? @ecx;G"<br>&gt;&gt; bp ntdll!RtlInitializeCriticalSectionAndSpinCount ".echo INICRIT;?<br>&gt;&gt; @ecx;G"<br>&gt;&gt; <br>&gt;&gt; Which resulted in the following output<br>&gt;&gt; ... Lots of other critical sections reported as INICRIT/DELCRIT ...<br>&gt;&gt; INICRIT<br>&gt;&gt; Evaluate expression: 63936328 = 0000000003cf9748
>> … Lots of other critical sections reported as INICRIT/DELCRIT …
>> DELCRIT
>> Evaluate expression: 63936328 = 00000000`03cf9748
>>
>> And then the AppVerifier complaint above.
>>
>>


Jakob B?hm, M.Sc.Eng. * xxxxx@danware.dk * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right.
Information in this posting may not be the official position of Danware
Data A/S, only the personal opinions of the author.

Well, the main thing is how you are resolving your imports. Avrf does not rewrite EATs, but instead just hooks IAT entries at image load time.

GetProcAddress is hooked through the static import hook mechanism, and this hook is used to provide future dynamic bound import hooks.

If you are manually traversing image EATs to resolve your imports, then you may not consistently get avrf hooks.

I seem to recall that all of the critsec kernel32 exports were plain forwarders, but it is possible that InitializeCriticalSection got turned into a thunk and not a forwarder. I can’t check that easily on my phone, but I’d imagine that you could make the determination yourself with the debugger pretty easily.

  • S

-----Original Message-----
From: Jakob Bohm
Sent: Wednesday, October 01, 2008 09:47
To: Kernel Debugging Interest List
Subject: Re:[windbg] AppVerifier complains about “freeing memory with Critical Section”, but I am sure I did DeleteCriticalSection()

Well, I am doing something more exotic, but strangely I do the same
for InitializeCriticalSection, in which case I wonder how avrf catches
one, but not the other.

One possibility though: If avrf catches imports of the ntdll
entrypoints, then InitializeCriticalSection() goes through a
front in another system DLL which does a normal static import of
ntdll!RtlInitializeCriticalSectionAndSpinCount(), while
DeleteCriticalSection() is a simple forwarder import causing my
code to call ntdll!RtlDeleteCriticalSection() directly.

Worth looking into though.

Skywing wrote:
> How are you retrieving the addresses of InitializeCriticalSection/LeaveCriticalSection?
>
>
> IIRC, avrf does a dllimport hook to arrange to be called when
> kernel32!InitializeCriticalSection/kernel32!DeleteCriticalSection, and
> these, among other things, insert/remove the crit sec into a tracking
> list that is scanned when memory is decommitted.
>
> Now, if you manage to directly call RtlDeleteCriticalSection in the
> delete path without going through the avrf hook, you may be able to
> trick avrf into thinking that you’ve got a leak when you do not really.
>
> Simple static dllimports and GetProcAddress should not result in this
> behavior, however, if you are doing something more exotic, then you
> might run into a situation similar to what you described.
>
> - S
>
>> From: Jakob Bohm
>> Sent: Wednesday, October 01, 2008 04:26
>> To: Kernel Debugging Interest List
>> Subject: [windbg] AppVerifier complains about “freeing memory with Critical Section”, but I am sure I did DeleteCriticalSection()
>>
>>
>> I suspect AppVerifier is wrong here, but I am not sure and would like
>> a second opinion, from someone on this lists who knows these parts of
>> AppVerifier more closely.
>>
>> OS is Windows 5.02 x64 uniprocessor workstation (aka XP x64) SP2
>> WinDbg is version 6.7.0005.1
>>
>> The scenario: A 64 bit service dynamically loads some “JIT-ed” code
>> into memory and runs it. After running for some time, the JIT-ed code
>> frees its resources and then the memory containing it is freed. Within
>> the allocated memory for the JIT-ed code/data are some
>> RTL_CRITICAL_SECTION objects initialized with
>> kernel32!InitializeCriticalSection and later deinitialized with
>> kernel32!DeleteCriticalSection . I have enabled AppVerifier for the
>> process with gflags.exe, and later attach to the service with WinDbg to
>> see and debug any AppVerifier induced messages.
>>
>> When the memory block containing the JIT-ed code and its data is freed,
>> AppVerifier complains like this:
>>
>> ===========================================================
>> VERIFIER STOP 0000000000000212: pid 0x4BC: releasing virtual memory
>> containing active critical section
>>
>> 0000000003CF9748 : Critical section address
>> 0000000000000000 : Initialization stack trace. Use dds to dump it if
>> non-NULL.
>> 0000000003CE0000 : Memory block address
>> 0000000000022000 : Memory block size
>> ===========================================================
>> This verifier stop is continuable.
>> After debugging it use go' to continue.<br>&gt;&gt; ===<br>&gt;&gt;<br>&gt;&gt; But I am almost sure I already deinitialized that critical section, in<br>&gt;&gt; particular:<br>&gt;&gt;<br>&gt;&gt; !locks -v does not list 3CE0000 as a known critical section<br>&gt;&gt;<br>&gt;&gt; A dump of address 3CE0000 shows that at least the first 16 bytes are 0<br>&gt;&gt;<br>&gt;&gt; Before the AppVerifier complaint I set the following 2 breakpoints<br>&gt;&gt; bp ntdll!RtlDeleteCriticalSection ".echo DELCRIT;? @ecx;G"<br>&gt;&gt; bp ntdll!RtlInitializeCriticalSectionAndSpinCount ".echo INICRIT;?<br>&gt;&gt; @ecx;G"<br>&gt;&gt;<br>&gt;&gt; Which resulted in the following output<br>&gt;&gt; ... Lots of other critical sections reported as INICRIT/DELCRIT ...<br>&gt;&gt; INICRIT<br>&gt;&gt; Evaluate expression: 63936328 = 0000000003cf9748
>> … Lots of other critical sections reported as INICRIT/DELCRIT …
>> DELCRIT
>> Evaluate expression: 63936328 = 00000000`03cf9748
>>
>> And then the AppVerifier complaint above.
>>
>>


Jakob B?hm, M.Sc.Eng. * xxxxx@danware.dk * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right.
Information in this posting may not be the official position of Danware
Data A/S, only the personal opinions of the author.


You are currently subscribed to windbg as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com