Intermittent Windows 2000 Crash

> Hi,

I am writing a KMDF driver using KMDF 1.7 and WDK 6001.18002. The
driver is designed to monitor process creation and deletion. As
processes create and terminate a callback is fired via registering
with pssetcreateprocessnotifyroutine() , details on these processes
are stored in a linked list in paged memory. Once an item is stored a
Kernel mode event is signaled which is registered in a user mode
application and the data is retrieved from the driver via an IO_CTRL.
This works fine and has been rigorously tested on every OS from
Windows 2000 through to Windows 7 and crashes intermittently on
Windows 2000.

It always crashes in the pssetcreateprocessnotifyroutine callback
function either adding items to a linked list, or freeing memory after
allocating, but the memory looks fine when I examine it in the
debugger. I check all return values from ExAllocatePoolWithTag and
FreePoolWithTag.

To load the driver I based my code on the NONPNP sample and use the
CreateService / startService method specified there.

The driver seems to crash only after the executable that starts it is
stopped prematurely( before it gets a chance to stop the driver
service or uninstall it), the driver then stopped manually using sc
service stop, and the executable to run it is then run immediately
several times in quick succession. The crash is always on allocating
memory or freeing it or occassionally clearing an event created in the
Kernel.

I have enclosed a crash dump with complete symbols below

As can be seen in the output there are complaints about .sys files
overlapping and I was wondering if this could be the cause of it? And
if so what could I do to resolve it?

My thought was that this is a CoInstaller issue as in a previous crash
I saw that the bottom of the callstack was WdfPreDeviceInstall().

If anyone could shed some light on this I would be eternally
grateful…

Output from Windbg follows:

**********************************************************************
*********
*
*
* Bugcheck Analysis
*
*
*
**********************************************************************
*********

Use !analyze -v to get detailed debugging information.

BugCheck A, {16, 2, 0, 80432306}

Loading symbols for f838e000 ProcessMon.sys -> ProcessMon.sys
**********************************************************************
***

Probably caused by : ProcessMon.sys ( ProcessMon!AddItem+7f )

Followup: MachineOwner

nt!RtlpNewSecurityObject+0x8f:
804554d8 cc int 3
kd> kn

ChildEBP RetAddr

00 edd5f6d0 8042afd6 nt!RtlpNewSecurityObject+0x8f
01 edd5f7c4 8041d4a2 nt!KdLogDbgPrint+0xdf
02 edd5f870 804686de nt!FsRtlOplockBreakToNone+0x113
03 edd5f8c4 8046006f nt!MiFreePoolPages+0x3f7
04 edd5f8f8 8045bbc9 nt!CmpGetRegistryValue+0x6d
05 edd5f98c 8046aad4 nt!PiProcessDriverInstance+0x169
06 edd5f9a0 8046aa2f nt!KiFindFirstSetRight+0x54
07 edd5f9ac 8052c808 nt!SeAliasBackupOpsSid+0x3
08 edd5fa98 f838f08f nt!IovCallDriver+0x64
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> .frame 09
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> !analyze -v
**********************************************************************
*********
*
*
* Bugcheck Analysis
*
*
*
**********************************************************************
*********

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid)
address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000016, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute
operation (only on chips which support this level of status)
Arg4: 80432306, address which referenced memory

OVERLAPPED_MODULE: Address regions for ‘vmx_fb’ and ‘VGA.dll’ overlap

READ_ADDRESS: 00000016 Nonpaged pool

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInitializeContextThread+10a
80432306 66837f1601 cmp word ptr [edi+16h],1

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8042afd6 to 804554d8

STACK_TEXT:
edd5f6d0 8042afd6 00000003 00000016 80432306
nt!RtlpNewSecurityObject+0x8f
edd5f7c4 8041d4a2 80450893 804d59a0 8049f9f1 nt!KdLogDbgPrint+0xdf
edd5f870 804686de 00002c00 000000d1 00000700
nt!FsRtlOplockBreakToNone+0x113
edd5f8c4 8046006f f87efaa4 00000030 8133b1d0 nt!MiFreePoolPages+0x3f7
edd5f8f8 8045bbc9 00000001 f87ef88c 00000000
nt!CmpGetRegistryValue+0x6d
edd5f98c 8046aad4 f87ef800 f87ef968 a5cbae90
nt!PiProcessDriverInstance+0x169
edd5f9a0 8046aa2f 00000000 00000000 8052c808
nt!KiFindFirstSetRight+0x54
edd5f9ac 8052c808 00000000 00000000 00000000
nt!SeAliasBackupOpsSid+0x3
edd5fa98 f838f08f 00000010 8052dc20 00000000 nt!IovCallDriver+0x64
edd5fab8 00000000 00000000 edd5fcb4 00000030 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]

STACK_COMMAND: kb

FOLLOWUP_IP:
ProcessMon!AddItem+7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
f838f08f 8b4dfc mov ecx,dword ptr [ebp-4]

FAULTING_SOURCE_CODE:
28: if (element != NULL) {
29:
30: // //now populate the structure
31: element->arraySize = iSize;
> 32: element->byteArray =
(BYTE*)ExAllocatePoolWithTag(PagedPool,element->arraySize,‘AdEb’);
33: //
34: // //ensure that the memory was
allocated properly
35: if(element->byteArray != NULL) {
36:
37: // //copy the memory that
we want into the array

SYMBOL_STACK_INDEX: 9

SYMBOL_NAME: ProcessMon!AddItem+7f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ProcessMon

IMAGE_NAME: ProcessMon.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4d9308ca

FAILURE_BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

Followup: MachineOwner

kd> lmv m kernel32
start end module name
7c570000 7c623000 KERNEL32 # (pdb symbols)
C:\WINDOWS\Symbols\Symbols_2000\dll\kernel32.pdb
Loaded symbol image file:
C:\WINDOWS\Symbols\Symbols_2000\dll\KERNEL32.dbg
Image path: C:\WINNT\system32\KERNEL32.dll
Image name: KERNEL32.dll
Timestamp: Thu Jan 06 09:17:41 2005 (41DD0235)
CheckSum: 000BCAE3
ImageSize: 000B3000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Kind Regards

Mark

Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.

I don't suppose anyone had any ideas on this did they?

Kind Regards

Mark


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Diggles
Sent: 30 March 2011 17:18
To: Windows System Software Devs Interest List
Subject: [ntdev] Intermittent Windows 2000 Crash

Hi,

I am writing a KMDF driver using KMDF 1.7 and WDK 6001.18002. The driver
is designed to monitor process creation and deletion. As processes
create and terminate a callback is fired via registering with
pssetcreateprocessnotifyroutine() , details on these processes are
stored in a linked list in paged memory. Once an item is stored a Kernel
mode event is signaled which is registered in a user mode application
and the data is retrieved from the driver via an IO_CTRL. This works
fine and has been rigorously tested on every OS from Windows 2000
through to Windows 7 and crashes intermittently on Windows 2000.

It always crashes in the pssetcreateprocessnotifyroutine callback
function either adding items to a linked list, or freeing memory after
allocating, but the memory looks fine when I examine it in the debugger.
I check all return values from ExAllocatePoolWithTag and
FreePoolWithTag.

To load the driver I based my code on the NONPNP sample and use the
CreateService / startService method specified there.

The driver seems to crash only after the executable that starts it is
stopped prematurely( before it gets a chance to stop the driver service
or uninstall it), the driver then stopped manually using sc service
stop, and the executable to run it is then run immediately several times
in quick succession. The crash is always on allocating memory or freeing
it or occassionally clearing an event created in the Kernel.

I have enclosed a crash dump with complete symbols below

As can be seen in the output there are complaints about .sys files
overlapping and I was wondering if this could be the cause of it? And if
so what could I do to resolve it?

My thought was that this is a CoInstaller issue as in a previous crash I
saw that the bottom of the callstack was WdfPreDeviceInstall().

If anyone could shed some light on this I would be eternally grateful..

Output from Windbg follows:

************************************************************************
*******
*
*
* Bugcheck Analysis
*
*
*
************************************************************************
*******

Use !analyze -v to get detailed debugging information.

BugCheck A, {16, 2, 0, 80432306}

Loading symbols for f838e000 ProcessMon.sys -> ProcessMon.sys
************************************************************************
*

Probably caused by : ProcessMon.sys ( ProcessMon!AddItem+7f )

Followup: MachineOwner

nt!RtlpNewSecurityObject+0x8f:
804554d8 cc int 3
kd> kn

ChildEBP RetAddr

00 edd5f6d0 8042afd6 nt!RtlpNewSecurityObject+0x8f
01 edd5f7c4 8041d4a2 nt!KdLogDbgPrint+0xdf
02 edd5f870 804686de nt!FsRtlOplockBreakToNone+0x113
03 edd5f8c4 8046006f nt!MiFreePoolPages+0x3f7
04 edd5f8f8 8045bbc9 nt!CmpGetRegistryValue+0x6d
05 edd5f98c 8046aad4 nt!PiProcessDriverInstance+0x169
06 edd5f9a0 8046aa2f nt!KiFindFirstSetRight+0x54
07 edd5f9ac 8052c808 nt!SeAliasBackupOpsSid+0x3
08 edd5fa98 f838f08f nt!IovCallDriver+0x64
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> .frame 09
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> !analyze -v
************************************************************************
*******
*
*
* Bugcheck Analysis
*
*
*
************************************************************************
*******

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000016, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute
operation (only on chips which support this level of status)

Arg4: 80432306, address which referenced memory

OVERLAPPED_MODULE: Address regions for 'vmx_fb' and 'VGA.dll' overlap

READ_ADDRESS: 00000016 Nonpaged pool

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInitializeContextThread+10a
80432306 66837f1601 cmp word ptr [edi+16h],1

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8042afd6 to 804554d8

STACK_TEXT:
edd5f6d0 8042afd6 00000003 00000016 80432306
nt!RtlpNewSecurityObject+0x8f
edd5f7c4 8041d4a2 80450893 804d59a0 8049f9f1 nt!KdLogDbgPrint+0xdf
edd5f870 804686de 00002c00 000000d1 00000700
nt!FsRtlOplockBreakToNone+0x113
edd5f8c4 8046006f f87efaa4 00000030 8133b1d0 nt!MiFreePoolPages+0x3f7
edd5f8f8 8045bbc9 00000001 f87ef88c 00000000 nt!CmpGetRegistryValue+0x6d

edd5f98c 8046aad4 f87ef800 f87ef968 a5cbae90
nt!PiProcessDriverInstance+0x169
edd5f9a0 8046aa2f 00000000 00000000 8052c808 nt!KiFindFirstSetRight+0x54

edd5f9ac 8052c808 00000000 00000000 00000000 nt!SeAliasBackupOpsSid+0x3
edd5fa98 f838f08f 00000010 8052dc20 00000000 nt!IovCallDriver+0x64
edd5fab8 00000000 00000000 edd5fcb4 00000030 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]

STACK_COMMAND: kb

FOLLOWUP_IP:
ProcessMon!AddItem+7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
f838f08f 8b4dfc mov ecx,dword ptr [ebp-4]

FAULTING_SOURCE_CODE:
28: if (element != NULL) {
29:
30: // //now populate the structure
31: element->arraySize = iSize;

32: element->byteArray =
(BYTE*)ExAllocatePoolWithTag(PagedPool,element->arraySize,'AdEb');

33: //
34: // //ensure that the memory was
allocated properly
35: if(element->byteArray != NULL) {

36:
37: // //copy the memory that
we want into the array

SYMBOL_STACK_INDEX: 9

SYMBOL_NAME: ProcessMon!AddItem+7f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ProcessMon

IMAGE_NAME: ProcessMon.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4d9308ca

FAILURE_BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

Followup: MachineOwner

kd> lmv m kernel32
start end module name
7c570000 7c623000 KERNEL32 # (pdb symbols)
C:\WINDOWS\Symbols\Symbols_2000\dll\kernel32.pdb
Loaded symbol image file:
C:\WINDOWS\Symbols\Symbols_2000\dll\KERNEL32.dbg
Image path: C:\WINNT\system32\KERNEL32.dll
Image name: KERNEL32.dll
Timestamp: Thu Jan 06 09:17:41 2005 (41DD0235)
CheckSum: 000BCAE3
ImageSize: 000B3000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Kind Regards

Mark


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:

To unsubscribe, visit the List Server section of OSR Online at

Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received
this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard
copy by an authorised signatory. The contents of this email may relate
to dealings with other companies within the Detica Limited group of
companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
England.

Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.

You’re at DISPATCH_LEVEL and attempting to access location 0x00000016 , which is typically caused by dereferencing a NULL pointer. However, the code passed the check for NULL, but that only means element was not NULL. It could have been anything from 1 to 0xFFFFFFFF. What is element? Is it a linked list and have you initialized it properly?

Gary G. Little

----- Original Message -----
From: “Mark Diggles”
To: “Windows System Software Devs Interest List”
Sent: Thursday, March 31, 2011 8:32:04 AM
Subject: RE: [ntdev] Intermittent Windows 2000 Crash

Intermittent Windows 2000 Crash
I don’t suppose anyone had any ideas on this did they?

Kind Regards

Mark

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Mark Diggles
Sent: 30 March 2011 17:18
To: Windows System Software Devs Interest List
Subject: [ntdev] Intermittent Windows 2000 Crash

Hi,

I am writing a KMDF driver using KMDF 1.7 and WDK 6001.18002. The driver is designed to monitor process creation and deletion. As processes create and terminate a callback is fired via registering with pssetcreateprocessnotifyroutine() , details on these processes are stored in a linked list in paged memory. Once an item is stored a Kernel mode event is signaled which is registered in a user mode application and the data is retrieved from the driver via an IO_CTRL. This works fine and has been rigorously tested on every OS from Windows 2000 through to Windows 7 and crashes intermittently on Windows 2000.

It always crashes in the pssetcreateprocessnotifyroutine callback function either adding items to a linked list, or freeing memory after allocating, but the memory looks fine when I examine it in the debugger. I check all return values from ExAllocatePoolWithTag and FreePoolWithTag.

To load the driver I based my code on the NONPNP sample and use the CreateService / startService method specified there.

The driver seems to crash only after the executable that starts it is stopped prematurely( before it gets a chance to stop the driver service or uninstall it), the driver then stopped manually using sc service stop, and the executable to run it is then run immediately several times in quick succession. The crash is always on allocating memory or freeing it or occassionally clearing an event created in the Kernel.

I have enclosed a crash dump with complete symbols below

As can be seen in the output there are complaints about .sys files overlapping and I was wondering if this could be the cause of it? And if so what could I do to resolve it?

My thought was that this is a CoInstaller issue as in a previous crash I saw that the bottom of the callstack was WdfPreDeviceInstall().

If anyone could shed some light on this I would be eternally grateful…

Output from Windbg follows:


*
Bugcheck Analysis *
*
***

Use !analyze -v to get detailed debugging information.

BugCheck A, {16, 2, 0, 80432306}

Loading symbols for f838e000 ProcessMon.sys -> ProcessMon.sys


Probably caused by : ProcessMon.sys ( ProcessMon!AddItem+7f )

Followup: MachineOwner
---------

nt!RtlpNewSecurityObject+0x8f:
804554d8 cc int 3
kd> kn
# ChildEBP RetAddr
00 edd5f6d0 8042afd6 nt!RtlpNewSecurityObject+0x8f
01 edd5f7c4 8041d4a2 nt!KdLogDbgPrint+0xdf
02 edd5f870 804686de nt!FsRtlOplockBreakToNone+0x113
03 edd5f8c4 8046006f nt!MiFreePoolPages+0x3f7
04 edd5f8f8 8045bbc9 nt!CmpGetRegistryValue+0x6d
05 edd5f98c 8046aad4 nt!PiProcessDriverInstance+0x169
06 edd5f9a0 8046aa2f nt!KiFindFirstSetRight+0x54
07 edd5f9ac 8052c808 nt!SeAliasBackupOpsSid+0x3
08 edd5fa98 f838f08f nt!IovCallDriver+0x64
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f [e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> .frame 09
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f [e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> !analyze -v
******
* *
* Bugcheck Analysis *
* *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000016, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)

Arg4: 80432306, address which referenced memory

OVERLAPPED_MODULE: Address regions for ‘vmx_fb’ and ‘VGA.dll’ overlap

READ_ADDRESS: 00000016 Nonpaged pool

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInitializeContextThread+10a
80432306 66837f1601 cmp word ptr [edi+16h],1

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8042afd6 to 804554d8

STACK_TEXT:
edd5f6d0 8042afd6 00000003 00000016 80432306 nt!RtlpNewSecurityObject+0x8f
edd5f7c4 8041d4a2 80450893 804d59a0 8049f9f1 nt!KdLogDbgPrint+0xdf
edd5f870 804686de 00002c00 000000d1 00000700 nt!FsRtlOplockBreakToNone+0x113
edd5f8c4 8046006f f87efaa4 00000030 8133b1d0 nt!MiFreePoolPages+0x3f7
edd5f8f8 8045bbc9 00000001 f87ef88c 00000000 nt!CmpGetRegistryValue+0x6d
edd5f98c 8046aad4 f87ef800 f87ef968 a5cbae90 nt!PiProcessDriverInstance+0x169
edd5f9a0 8046aa2f 00000000 00000000 8052c808 nt!KiFindFirstSetRight+0x54
edd5f9ac 8052c808 00000000 00000000 00000000 nt!SeAliasBackupOpsSid+0x3
edd5fa98 f838f08f 00000010 8052dc20 00000000 nt!IovCallDriver+0x64
edd5fab8 00000000 00000000 edd5fcb4 00000030 ProcessMon!AddItem+0x7f [e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]

STACK_COMMAND: kb

FOLLOWUP_IP:
ProcessMon!AddItem+7f [e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
f838f08f 8b4dfc mov ecx,dword ptr [ebp-4]

FAULTING_SOURCE_CODE:
28: if (element != NULL) {
29:
30: // //now populate the structure
31: element->arraySize = iSize;
> 32: element->byteArray = (BYTE*)ExAllocatePoolWithTag(PagedPool,element->arraySize,‘AdEb’);

33: //
34: // //ensure that the memory was allocated properly
35: if(element->byteArray != NULL) {
36:
37: // //copy the memory that we want into the array

SYMBOL_STACK_INDEX: 9

SYMBOL_NAME: ProcessMon!AddItem+7f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ProcessMon

IMAGE_NAME: ProcessMon.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4d9308ca

FAILURE_BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

Followup: MachineOwner
---------

kd> lmv m kernel32
start end module name
7c570000 7c623000 KERNEL32 # (pdb symbols) C:\WINDOWS\Symbols\Symbols_2000\dll\kernel32.pdb
Loaded symbol image file: C:\WINDOWS\Symbols\Symbols_2000\dll\KERNEL32.dbg
Image path: C:\WINNT\system32\KERNEL32.dll
Image name: KERNEL32.dll
Timestamp: Thu Jan 06 09:17:41 2005 (41DD0235)
CheckSum: 000BCAE3
ImageSize: 000B3000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Kind Regards

Mark

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.

Intermittent Windows 2000 CrashUnfortunately your symbols are messed up, so
the call stack doesn’t make any sense. The crash is a NULL pointer
dereference, outside of that it’s hard to say without correct symbols.

Have you tried testing this under Verifier? Could be a pool corruption of
some sort.

-scott


Scott Noone
Consulting Associate and Chief System Problem Analyst
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Mark Diggles” wrote in message
news:xxxxx@ntdev…
I don’t suppose anyone had any ideas on this did they?

Kind Regards

Mark

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Diggles
Sent: 30 March 2011 17:18
To: Windows System Software Devs Interest List
Subject: [ntdev] Intermittent Windows 2000 Crash

Hi,

I am writing a KMDF driver using KMDF 1.7 and WDK 6001.18002. The driver is
designed to monitor process creation and deletion. As processes create and
terminate a callback is fired via registering with
pssetcreateprocessnotifyroutine() , details on these processes are stored in
a linked list in paged memory. Once an item is stored a Kernel mode event is
signaled which is registered in a user mode application and the data is
retrieved from the driver via an IO_CTRL. This works fine and has been
rigorously tested on every OS from Windows 2000 through to Windows 7 and
crashes intermittently on Windows 2000.

It always crashes in the pssetcreateprocessnotifyroutine callback function
either adding items to a linked list, or freeing memory after allocating,
but the memory looks fine when I examine it in the debugger. I check all
return values from ExAllocatePoolWithTag and FreePoolWithTag.

To load the driver I based my code on the NONPNP sample and use the
CreateService / startService method specified there.

The driver seems to crash only after the executable that starts it is
stopped prematurely( before it gets a chance to stop the driver service or
uninstall it), the driver then stopped manually using sc service stop, and
the executable to run it is then run immediately several times in quick
succession. The crash is always on allocating memory or freeing it or
occassionally clearing an event created in the Kernel.

I have enclosed a crash dump with complete symbols below

As can be seen in the output there are complaints about .sys files
overlapping and I was wondering if this could be the cause of it? And if so
what could I do to resolve it?

My thought was that this is a CoInstaller issue as in a previous crash I saw
that the bottom of the callstack was WdfPreDeviceInstall().

If anyone could shed some light on this I would be eternally grateful…

Output from Windbg follows:




Bugcheck Analysis



******

Use !analyze -v to get detailed debugging information.

BugCheck A, {16, 2, 0, 80432306}

Loading symbols for f838e000 ProcessMon.sys -> ProcessMon.sys


Probably caused by : ProcessMon.sys ( ProcessMon!AddItem+7f )

Followup: MachineOwner
---------

nt!RtlpNewSecurityObject+0x8f:
804554d8 cc int 3
kd> kn
# ChildEBP RetAddr
00 edd5f6d0 8042afd6 nt!RtlpNewSecurityObject+0x8f
01 edd5f7c4 8041d4a2 nt!KdLogDbgPrint+0xdf
02 edd5f870 804686de nt!FsRtlOplockBreakToNone+0x113
03 edd5f8c4 8046006f nt!MiFreePoolPages+0x3f7
04 edd5f8f8 8045bbc9 nt!CmpGetRegistryValue+0x6d
05 edd5f98c 8046aad4 nt!PiProcessDriverInstance+0x169
06 edd5f9a0 8046aa2f nt!KiFindFirstSetRight+0x54
07 edd5f9ac 8052c808 nt!SeAliasBackupOpsSid+0x3
08 edd5fa98 f838f08f nt!IovCallDriver+0x64
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> .frame 09
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> !analyze -v



Bugcheck Analysis



*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at
an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000016, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation
(only on chips which support this level of status)

Arg4: 80432306, address which referenced memory

OVERLAPPED_MODULE: Address regions for ‘vmx_fb’ and ‘VGA.dll’ overlap

READ_ADDRESS: 00000016 Nonpaged pool

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInitializeContextThread+10a
80432306 66837f1601 cmp word ptr [edi+16h],1

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8042afd6 to 804554d8

STACK_TEXT:
edd5f6d0 8042afd6 00000003 00000016 80432306 nt!RtlpNewSecurityObject+0x8f
edd5f7c4 8041d4a2 80450893 804d59a0 8049f9f1 nt!KdLogDbgPrint+0xdf
edd5f870 804686de 00002c00 000000d1 00000700 nt!FsRtlOplockBreakToNone+0x113
edd5f8c4 8046006f f87efaa4 00000030 8133b1d0 nt!MiFreePoolPages+0x3f7
edd5f8f8 8045bbc9 00000001 f87ef88c 00000000 nt!CmpGetRegistryValue+0x6d
edd5f98c 8046aad4 f87ef800 f87ef968 a5cbae90
nt!PiProcessDriverInstance+0x169
edd5f9a0 8046aa2f 00000000 00000000 8052c808 nt!KiFindFirstSetRight+0x54
edd5f9ac 8052c808 00000000 00000000 00000000 nt!SeAliasBackupOpsSid+0x3
edd5fa98 f838f08f 00000010 8052dc20 00000000 nt!IovCallDriver+0x64
edd5fab8 00000000 00000000 edd5fcb4 00000030 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]

STACK_COMMAND: kb

FOLLOWUP_IP:
ProcessMon!AddItem+7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
f838f08f 8b4dfc mov ecx,dword ptr [ebp-4]

FAULTING_SOURCE_CODE:
28: if (element != NULL) {
29:
30: // //now populate the structure
31: element->arraySize = iSize;
> 32: element->byteArray =
> (BYTE*)ExAllocatePoolWithTag(PagedPool,element->arraySize,‘AdEb’);

33: //
34: // //ensure that the memory was
allocated properly
35: if(element->byteArray != NULL) {
36:
37: // //copy the memory that we
want into the array

SYMBOL_STACK_INDEX: 9

SYMBOL_NAME: ProcessMon!AddItem+7f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ProcessMon

IMAGE_NAME: ProcessMon.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4d9308ca

FAILURE_BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

Followup: MachineOwner
---------

kd> lmv m kernel32
start end module name
7c570000 7c623000 KERNEL32 # (pdb symbols)
C:\WINDOWS\Symbols\Symbols_2000\dll\kernel32.pdb
Loaded symbol image file:
C:\WINDOWS\Symbols\Symbols_2000\dll\KERNEL32.dbg
Image path: C:\WINNT\system32\KERNEL32.dll
Image name: KERNEL32.dll
Timestamp: Thu Jan 06 09:17:41 2005 (41DD0235)
CheckSum: 000BCAE3
ImageSize: 000B3000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Kind Regards

Mark


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer Please consider the
environment before printing this email. This message should be regarded as
confidential. If you have received this email in error please notify the
sender and destroy it immediately. Statements of intent shall only become
binding when confirmed in hard copy by an authorised signatory. The contents
of this email may relate to dealings with other companies within the Detica
Limited group of companies. Detica Limited is registered in England under
No: 1337451. Registered offices: Surrey Research Park, Guildford, Surrey,
GU2 7YP, England. Please consider the environment before printing this
email. This message should be regarded as confidential. If you have received
this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory. The contents of this email may relate to
dealings with other companies within the Detica Limited group of companies.
Detica Limited is registered in England under No: 1337451. Registered
offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.

I have been running it under driver verifier this morning, and have
managed to update symbols, it seems that its crashing on KeSetEvent and
after placing a memory breakpoint this area is null.

I think that its something to with my creating a notification event in
the driver that is waited on in a user app and once this app is
destroyed prematurely something nasty is happening to the kernel event.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: 31 March 2011 15:37
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Intermittent Windows 2000 Crash

Intermittent Windows 2000 CrashUnfortunately your symbols are messed up,
so the call stack doesn’t make any sense. The crash is a NULL pointer
dereference, outside of that it’s hard to say without correct symbols.

Have you tried testing this under Verifier? Could be a pool corruption
of some sort.

-scott


Scott Noone
Consulting Associate and Chief System Problem Analyst OSR Open Systems
Resources, Inc.
http://www.osronline.com

“Mark Diggles” wrote in message
news:xxxxx@ntdev…
I don’t suppose anyone had any ideas on this did they?

Kind Regards

Mark

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Diggles
Sent: 30 March 2011 17:18
To: Windows System Software Devs Interest List
Subject: [ntdev] Intermittent Windows 2000 Crash

Hi,

I am writing a KMDF driver using KMDF 1.7 and WDK 6001.18002. The driver
is
designed to monitor process creation and deletion. As processes create
and
terminate a callback is fired via registering with
pssetcreateprocessnotifyroutine() , details on these processes are
stored in
a linked list in paged memory. Once an item is stored a Kernel mode
event is
signaled which is registered in a user mode application and the data is
retrieved from the driver via an IO_CTRL. This works fine and has been
rigorously tested on every OS from Windows 2000 through to Windows 7 and

crashes intermittently on Windows 2000.

It always crashes in the pssetcreateprocessnotifyroutine callback
function
either adding items to a linked list, or freeing memory after
allocating,
but the memory looks fine when I examine it in the debugger. I check all

return values from ExAllocatePoolWithTag and FreePoolWithTag.

To load the driver I based my code on the NONPNP sample and use the
CreateService / startService method specified there.

The driver seems to crash only after the executable that starts it is
stopped prematurely( before it gets a chance to stop the driver service
or
uninstall it), the driver then stopped manually using sc service stop,
and
the executable to run it is then run immediately several times in quick
succession. The crash is always on allocating memory or freeing it or
occassionally clearing an event created in the Kernel.

I have enclosed a crash dump with complete symbols below

As can be seen in the output there are complaints about .sys files
overlapping and I was wondering if this could be the cause of it? And if
so
what could I do to resolve it?

My thought was that this is a CoInstaller issue as in a previous crash I
saw
that the bottom of the callstack was WdfPreDeviceInstall().

If anyone could shed some light on this I would be eternally grateful…

Output from Windbg follows:





Bugcheck Analysis






Use !analyze -v to get detailed debugging information.

BugCheck A, {16, 2, 0, 80432306}

Loading symbols for f838e000 ProcessMon.sys -> ProcessMon.sys



Probably caused by : ProcessMon.sys ( ProcessMon!AddItem+7f )

Followup: MachineOwner
---------

nt!RtlpNewSecurityObject+0x8f:
804554d8 cc int 3
kd> kn
# ChildEBP RetAddr
00 edd5f6d0 8042afd6 nt!RtlpNewSecurityObject+0x8f
01 edd5f7c4 8041d4a2 nt!KdLogDbgPrint+0xdf
02 edd5f870 804686de nt!FsRtlOplockBreakToNone+0x113
03 edd5f8c4 8046006f nt!MiFreePoolPages+0x3f7
04 edd5f8f8 8045bbc9 nt!CmpGetRegistryValue+0x6d
05 edd5f98c 8046aad4 nt!PiProcessDriverInstance+0x169
06 edd5f9a0 8046aa2f nt!KiFindFirstSetRight+0x54
07 edd5f9ac 8052c808 nt!SeAliasBackupOpsSid+0x3
08 edd5fa98 f838f08f nt!IovCallDriver+0x64
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> .frame 09
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> !analyze -v




Bugcheck Analysis



**************************************************
*******

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at
an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000016, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute
operation
(only on chips which support this level of status)

Arg4: 80432306, address which referenced memory

OVERLAPPED_MODULE: Address regions for ‘vmx_fb’ and ‘VGA.dll’ overlap

READ_ADDRESS: 00000016 Nonpaged pool

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInitializeContextThread+10a
80432306 66837f1601 cmp word ptr [edi+16h],1

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8042afd6 to 804554d8

STACK_TEXT:
edd5f6d0 8042afd6 00000003 00000016 80432306
nt!RtlpNewSecurityObject+0x8f
edd5f7c4 8041d4a2 80450893 804d59a0 8049f9f1 nt!KdLogDbgPrint+0xdf
edd5f870 804686de 00002c00 000000d1 00000700
nt!FsRtlOplockBreakToNone+0x113
edd5f8c4 8046006f f87efaa4 00000030 8133b1d0 nt!MiFreePoolPages+0x3f7
edd5f8f8 8045bbc9 00000001 f87ef88c 00000000 nt!CmpGetRegistryValue+0x6d
edd5f98c 8046aad4 f87ef800 f87ef968 a5cbae90
nt!PiProcessDriverInstance+0x169
edd5f9a0 8046aa2f 00000000 00000000 8052c808 nt!KiFindFirstSetRight+0x54
edd5f9ac 8052c808 00000000 00000000 00000000 nt!SeAliasBackupOpsSid+0x3
edd5fa98 f838f08f 00000010 8052dc20 00000000 nt!IovCallDriver+0x64
edd5fab8 00000000 00000000 edd5fcb4 00000030 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]

STACK_COMMAND: kb

FOLLOWUP_IP:
ProcessMon!AddItem+7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
f838f08f 8b4dfc mov ecx,dword ptr [ebp-4]

FAULTING_SOURCE_CODE:
28: if (element != NULL) {
29:
30: // //now populate the structure
31: element->arraySize = iSize;
> 32: element->byteArray =
> (BYTE*)ExAllocatePoolWithTag(PagedPool,element->arraySize,‘AdEb’);

33: //
34: // //ensure that the memory was
allocated properly
35: if(element->byteArray != NULL) {
36:
37: // //copy the memory that
we
want into the array

SYMBOL_STACK_INDEX: 9

SYMBOL_NAME: ProcessMon!AddItem+7f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ProcessMon

IMAGE_NAME: ProcessMon.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4d9308ca

FAILURE_BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

Followup: MachineOwner
---------

kd> lmv m kernel32
start end module name
7c570000 7c623000 KERNEL32 # (pdb symbols)
C:\WINDOWS\Symbols\Symbols_2000\dll\kernel32.pdb
Loaded symbol image file:
C:\WINDOWS\Symbols\Symbols_2000\dll\KERNEL32.dbg
Image path: C:\WINNT\system32\KERNEL32.dll
Image name: KERNEL32.dll
Timestamp: Thu Jan 06 09:17:41 2005 (41DD0235)
CheckSum: 000BCAE3
ImageSize: 000B3000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Kind Regards

Mark


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer Please consider the
environment before printing this email. This message should be regarded
as
confidential. If you have received this email in error please notify the

sender and destroy it immediately. Statements of intent shall only
become
binding when confirmed in hard copy by an authorised signatory. The
contents
of this email may relate to dealings with other companies within the
Detica
Limited group of companies. Detica Limited is registered in England
under
No: 1337451. Registered offices: Surrey Research Park, Guildford,
Surrey,
GU2 7YP, England. Please consider the environment before printing this
email. This message should be regarded as confidential. If you have
received
this email in error please notify the sender and destroy it immediately.

Statements of intent shall only become binding when confirmed in hard
copy
by an authorised signatory. The contents of this email may relate to
dealings with other companies within the Detica Limited group of
companies.
Detica Limited is registered in England under No: 1337451. Registered
offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.

You’re not calling KeSetEvent at IRQL DISPATCH_LEVEL are you? This is a common problem… it’s not allowed, strictly speaking.

Peter
OSR

No, I have set ExecutionLevel to WdfExecutionLevelPassive so that should
not be possible, it looks like the memory is corrupt.

Running more tests now

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@osr.com
Sent: 31 March 2011 16:31
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Intermittent Windows 2000 Crash

You’re not calling KeSetEvent at IRQL DISPATCH_LEVEL are you? This is a
common problem… it’s not allowed, strictly speaking.

Peter
OSR


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.

ahh … there is part of the problem: sharing events between a driver and an application. That’s been discussed here MANY times, and typically discouraged. It would be best to simply post an IO request from the application to the driver, possibly a locally defined an IOCTL. When the app goes away the request gets cancelled and the driver handles it properly … ok … SHOULD handle it properly, but given its KMDF it’s a good possibility. when the event happens that would have signaled the shared event the driver simply completes the IO request.

Same affect as a shared event, and MUCH safer.

Gary G. Little

----- Original Message -----
From: “Mark Diggles”
To: “Windows System Software Devs Interest List”
Sent: Thursday, March 31, 2011 9:45:26 AM
Subject: RE: [ntdev] Intermittent Windows 2000 Crash

I have been running it under driver verifier this morning, and have
managed to update symbols, it seems that its crashing on KeSetEvent and
after placing a memory breakpoint this area is null.

I think that its something to with my creating a notification event in
the driver that is waited on in a user app and once this app is
destroyed prematurely something nasty is happening to the kernel event.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: 31 March 2011 15:37
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Intermittent Windows 2000 Crash

Intermittent Windows 2000 CrashUnfortunately your symbols are messed up,
so the call stack doesn’t make any sense. The crash is a NULL pointer
dereference, outside of that it’s hard to say without correct symbols.

Have you tried testing this under Verifier? Could be a pool corruption
of some sort.

-scott


Scott Noone
Consulting Associate and Chief System Problem Analyst OSR Open Systems
Resources, Inc.
http://www.osronline.com

“Mark Diggles” wrote in message
news:xxxxx@ntdev…
I don’t suppose anyone had any ideas on this did they?

Kind Regards

Mark

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Diggles
Sent: 30 March 2011 17:18
To: Windows System Software Devs Interest List
Subject: [ntdev] Intermittent Windows 2000 Crash

Hi,

I am writing a KMDF driver using KMDF 1.7 and WDK 6001.18002. The driver
is
designed to monitor process creation and deletion. As processes create
and
terminate a callback is fired via registering with
pssetcreateprocessnotifyroutine() , details on these processes are
stored in
a linked list in paged memory. Once an item is stored a Kernel mode
event is
signaled which is registered in a user mode application and the data is
retrieved from the driver via an IO_CTRL. This works fine and has been
rigorously tested on every OS from Windows 2000 through to Windows 7 and

crashes intermittently on Windows 2000.

It always crashes in the pssetcreateprocessnotifyroutine callback
function
either adding items to a linked list, or freeing memory after
allocating,
but the memory looks fine when I examine it in the debugger. I check all

return values from ExAllocatePoolWithTag and FreePoolWithTag.

To load the driver I based my code on the NONPNP sample and use the
CreateService / startService method specified there.

The driver seems to crash only after the executable that starts it is
stopped prematurely( before it gets a chance to stop the driver service
or
uninstall it), the driver then stopped manually using sc service stop,
and
the executable to run it is then run immediately several times in quick
succession. The crash is always on allocating memory or freeing it or
occassionally clearing an event created in the Kernel.

I have enclosed a crash dump with complete symbols below

As can be seen in the output there are complaints about .sys files
overlapping and I was wondering if this could be the cause of it? And if
so
what could I do to resolve it?

My thought was that this is a CoInstaller issue as in a previous crash I
saw
that the bottom of the callstack was WdfPreDeviceInstall().

If anyone could shed some light on this I would be eternally grateful…

Output from Windbg follows:





Bugcheck Analysis






Use !analyze -v to get detailed debugging information.

BugCheck A, {16, 2, 0, 80432306}

Loading symbols for f838e000 ProcessMon.sys -> ProcessMon.sys



Probably caused by : ProcessMon.sys ( ProcessMon!AddItem+7f )

Followup: MachineOwner
---------

nt!RtlpNewSecurityObject+0x8f:
804554d8 cc int 3
kd> kn
# ChildEBP RetAddr
00 edd5f6d0 8042afd6 nt!RtlpNewSecurityObject+0x8f
01 edd5f7c4 8041d4a2 nt!KdLogDbgPrint+0xdf
02 edd5f870 804686de nt!FsRtlOplockBreakToNone+0x113
03 edd5f8c4 8046006f nt!MiFreePoolPages+0x3f7
04 edd5f8f8 8045bbc9 nt!CmpGetRegistryValue+0x6d
05 edd5f98c 8046aad4 nt!PiProcessDriverInstance+0x169
06 edd5f9a0 8046aa2f nt!KiFindFirstSetRight+0x54
07 edd5f9ac 8052c808 nt!SeAliasBackupOpsSid+0x3
08 edd5fa98 f838f08f nt!IovCallDriver+0x64
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> .frame 09
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> !analyze -v
*********************



Bugcheck Analysis



***********************************************************************
*******

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at
an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000016, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute
operation
(only on chips which support this level of status)

Arg4: 80432306, address which referenced memory

OVERLAPPED_MODULE: Address regions for ‘vmx_fb’ and ‘VGA.dll’ overlap

READ_ADDRESS: 00000016 Nonpaged pool

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInitializeContextThread+10a
80432306 66837f1601 cmp word ptr [edi+16h],1

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8042afd6 to 804554d8

STACK_TEXT:
edd5f6d0 8042afd6 00000003 00000016 80432306
nt!RtlpNewSecurityObject+0x8f
edd5f7c4 8041d4a2 80450893 804d59a0 8049f9f1 nt!KdLogDbgPrint+0xdf
edd5f870 804686de 00002c00 000000d1 00000700
nt!FsRtlOplockBreakToNone+0x113
edd5f8c4 8046006f f87efaa4 00000030 8133b1d0 nt!MiFreePoolPages+0x3f7
edd5f8f8 8045bbc9 00000001 f87ef88c 00000000 nt!CmpGetRegistryValue+0x6d
edd5f98c 8046aad4 f87ef800 f87ef968 a5cbae90
nt!PiProcessDriverInstance+0x169
edd5f9a0 8046aa2f 00000000 00000000 8052c808 nt!KiFindFirstSetRight+0x54
edd5f9ac 8052c808 00000000 00000000 00000000 nt!SeAliasBackupOpsSid+0x3
edd5fa98 f838f08f 00000010 8052dc20 00000000 nt!IovCallDriver+0x64
edd5fab8 00000000 00000000 edd5fcb4 00000030 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]

STACK_COMMAND: kb

FOLLOWUP_IP:
ProcessMon!AddItem+7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
f838f08f 8b4dfc mov ecx,dword ptr [ebp-4]

FAULTING_SOURCE_CODE:
28: if (element != NULL) {
29:
30: // //now populate the structure
31: element->arraySize = iSize;
> 32: element->byteArray =
> (BYTE*)ExAllocatePoolWithTag(PagedPool,element->arraySize,‘AdEb’);

33: //
34: // //ensure that the memory was
allocated properly
35: if(element->byteArray != NULL) {
36:
37: // //copy the memory that
we
want into the array

SYMBOL_STACK_INDEX: 9

SYMBOL_NAME: ProcessMon!AddItem+7f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ProcessMon

IMAGE_NAME: ProcessMon.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4d9308ca

FAILURE_BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

Followup: MachineOwner
---------

kd> lmv m kernel32
start end module name
7c570000 7c623000 KERNEL32 # (pdb symbols)
C:\WINDOWS\Symbols\Symbols_2000\dll\kernel32.pdb
Loaded symbol image file:
C:\WINDOWS\Symbols\Symbols_2000\dll\KERNEL32.dbg
Image path: C:\WINNT\system32\KERNEL32.dll
Image name: KERNEL32.dll
Timestamp: Thu Jan 06 09:17:41 2005 (41DD0235)
CheckSum: 000BCAE3
ImageSize: 000B3000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Kind Regards

Mark


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer Please consider the
environment before printing this email. This message should be regarded
as
confidential. If you have received this email in error please notify the

sender and destroy it immediately. Statements of intent shall only
become
binding when confirmed in hard copy by an authorised signatory. The
contents
of this email may relate to dealings with other companies within the
Detica
Limited group of companies. Detica Limited is registered in England
under
No: 1337451. Registered offices: Surrey Research Park, Guildford,
Surrey,
GU2 7YP, England. Please consider the environment before printing this
email. This message should be regarded as confidential. If you have
received
this email in error please notify the sender and destroy it immediately.

Statements of intent shall only become binding when confirmed in hard
copy
by an authorised signatory. The contents of this email may relate to
dealings with other companies within the Detica Limited group of
companies.
Detica Limited is registered in England under No: 1337451. Registered
offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

That’s great, thanks for the information, I had not thought of doing it
that way, I will give that a try,

Thank-you very much everyone your help has been invaluable


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gary G. Little
Sent: 31 March 2011 16:37
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Intermittent Windows 2000 Crash

ahh … there is part of the problem: sharing events between a driver
and an application. That’s been discussed here MANY times, and typically
discouraged. It would be best to simply post an IO request from the
application to the driver, possibly a locally defined an IOCTL. When the
app goes away the request gets cancelled and the driver handles it
properly … ok … SHOULD handle it properly, but given its KMDF it’s a
good possibility. when the event happens that would have signaled the
shared event the driver simply completes the IO request.

Same affect as a shared event, and MUCH safer.

Gary G. Little

----- Original Message -----
From: “Mark Diggles”
To: “Windows System Software Devs Interest List”
Sent: Thursday, March 31, 2011 9:45:26 AM
Subject: RE: [ntdev] Intermittent Windows 2000 Crash

I have been running it under driver verifier this morning, and have
managed to update symbols, it seems that its crashing on KeSetEvent and
after placing a memory breakpoint this area is null.

I think that its something to with my creating a notification event in
the driver that is waited on in a user app and once this app is
destroyed prematurely something nasty is happening to the kernel event.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: 31 March 2011 15:37
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Intermittent Windows 2000 Crash

Intermittent Windows 2000 CrashUnfortunately your symbols are messed up,
so the call stack doesn’t make any sense. The crash is a NULL pointer
dereference, outside of that it’s hard to say without correct symbols.

Have you tried testing this under Verifier? Could be a pool corruption
of some sort.

-scott


Scott Noone
Consulting Associate and Chief System Problem Analyst OSR Open Systems
Resources, Inc.
http://www.osronline.com

“Mark Diggles” wrote in message
news:xxxxx@ntdev…
I don’t suppose anyone had any ideas on this did they?

Kind Regards

Mark

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Diggles
Sent: 30 March 2011 17:18
To: Windows System Software Devs Interest List
Subject: [ntdev] Intermittent Windows 2000 Crash

Hi,

I am writing a KMDF driver using KMDF 1.7 and WDK 6001.18002. The driver
is
designed to monitor process creation and deletion. As processes create
and
terminate a callback is fired via registering with
pssetcreateprocessnotifyroutine() , details on these processes are
stored in
a linked list in paged memory. Once an item is stored a Kernel mode
event is
signaled which is registered in a user mode application and the data is
retrieved from the driver via an IO_CTRL. This works fine and has been
rigorously tested on every OS from Windows 2000 through to Windows 7 and

crashes intermittently on Windows 2000.

It always crashes in the pssetcreateprocessnotifyroutine callback
function
either adding items to a linked list, or freeing memory after
allocating,
but the memory looks fine when I examine it in the debugger. I check all

return values from ExAllocatePoolWithTag and FreePoolWithTag.

To load the driver I based my code on the NONPNP sample and use the
CreateService / startService method specified there.

The driver seems to crash only after the executable that starts it is
stopped prematurely( before it gets a chance to stop the driver service
or
uninstall it), the driver then stopped manually using sc service stop,
and
the executable to run it is then run immediately several times in quick
succession. The crash is always on allocating memory or freeing it or
occassionally clearing an event created in the Kernel.

I have enclosed a crash dump with complete symbols below

As can be seen in the output there are complaints about .sys files
overlapping and I was wondering if this could be the cause of it? And if
so
what could I do to resolve it?

My thought was that this is a CoInstaller issue as in a previous crash I
saw
that the bottom of the callstack was WdfPreDeviceInstall().

If anyone could shed some light on this I would be eternally grateful…

Output from Windbg follows:





Bugcheck Analysis






Use !analyze -v to get detailed debugging information.

BugCheck A, {16, 2, 0, 80432306}

Loading symbols for f838e000 ProcessMon.sys -> ProcessMon.sys



Probably caused by : ProcessMon.sys ( ProcessMon!AddItem+7f )

Followup: MachineOwner
---------

nt!RtlpNewSecurityObject+0x8f:
804554d8 cc int 3
kd> kn
# ChildEBP RetAddr
00 edd5f6d0 8042afd6 nt!RtlpNewSecurityObject+0x8f
01 edd5f7c4 8041d4a2 nt!KdLogDbgPrint+0xdf
02 edd5f870 804686de nt!FsRtlOplockBreakToNone+0x113
03 edd5f8c4 8046006f nt!MiFreePoolPages+0x3f7
04 edd5f8f8 8045bbc9 nt!CmpGetRegistryValue+0x6d
05 edd5f98c 8046aad4 nt!PiProcessDriverInstance+0x169
06 edd5f9a0 8046aa2f nt!KiFindFirstSetRight+0x54
07 edd5f9ac 8052c808 nt!SeAliasBackupOpsSid+0x3
08 edd5fa98 f838f08f nt!IovCallDriver+0x64
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> .frame 09
09 edd5fab8 00000000 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
kd> !analyze -v




Bugcheck Analysis



**************************************************
*******

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at
an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000016, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute
operation
(only on chips which support this level of status)

Arg4: 80432306, address which referenced memory

OVERLAPPED_MODULE: Address regions for ‘vmx_fb’ and ‘VGA.dll’ overlap

READ_ADDRESS: 00000016 Nonpaged pool

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInitializeContextThread+10a
80432306 66837f1601 cmp word ptr [edi+16h],1

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8042afd6 to 804554d8

STACK_TEXT:
edd5f6d0 8042afd6 00000003 00000016 80432306
nt!RtlpNewSecurityObject+0x8f
edd5f7c4 8041d4a2 80450893 804d59a0 8049f9f1 nt!KdLogDbgPrint+0xdf
edd5f870 804686de 00002c00 000000d1 00000700
nt!FsRtlOplockBreakToNone+0x113
edd5f8c4 8046006f f87efaa4 00000030 8133b1d0 nt!MiFreePoolPages+0x3f7
edd5f8f8 8045bbc9 00000001 f87ef88c 00000000 nt!CmpGetRegistryValue+0x6d
edd5f98c 8046aad4 f87ef800 f87ef968 a5cbae90
nt!PiProcessDriverInstance+0x169
edd5f9a0 8046aa2f 00000000 00000000 8052c808 nt!KiFindFirstSetRight+0x54
edd5f9ac 8052c808 00000000 00000000 00000000 nt!SeAliasBackupOpsSid+0x3
edd5fa98 f838f08f 00000010 8052dc20 00000000 nt!IovCallDriver+0x64
edd5fab8 00000000 00000000 edd5fcb4 00000030 ProcessMon!AddItem+0x7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]

STACK_COMMAND: kb

FOLLOWUP_IP:
ProcessMon!AddItem+7f
[e:\dev\hostagent\agent\drvprocessmonitoring\linkedlist.c @ 32]
f838f08f 8b4dfc mov ecx,dword ptr [ebp-4]

FAULTING_SOURCE_CODE:
28: if (element != NULL) {
29:
30: // //now populate the structure
31: element->arraySize = iSize;
> 32: element->byteArray =
> (BYTE*)ExAllocatePoolWithTag(PagedPool,element->arraySize,‘AdEb’);

33: //
34: // //ensure that the memory was
allocated properly
35: if(element->byteArray != NULL) {
36:
37: // //copy the memory that
we
want into the array

SYMBOL_STACK_INDEX: 9

SYMBOL_NAME: ProcessMon!AddItem+7f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ProcessMon

IMAGE_NAME: ProcessMon.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4d9308ca

FAILURE_BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

BUCKET_ID: 0xA_VRFK_ProcessMon!AddItem+7f

Followup: MachineOwner
---------

kd> lmv m kernel32
start end module name
7c570000 7c623000 KERNEL32 # (pdb symbols)
C:\WINDOWS\Symbols\Symbols_2000\dll\kernel32.pdb
Loaded symbol image file:
C:\WINDOWS\Symbols\Symbols_2000\dll\KERNEL32.dbg
Image path: C:\WINNT\system32\KERNEL32.dll
Image name: KERNEL32.dll
Timestamp: Thu Jan 06 09:17:41 2005 (41DD0235)
CheckSum: 000BCAE3
ImageSize: 000B3000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Kind Regards

Mark


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer Please consider the
environment before printing this email. This message should be regarded
as
confidential. If you have received this email in error please notify the

sender and destroy it immediately. Statements of intent shall only
become
binding when confirmed in hard copy by an authorised signatory. The
contents
of this email may relate to dealings with other companies within the
Detica
Limited group of companies. Detica Limited is registered in England
under
No: 1337451. Registered offices: Surrey Research Park, Guildford,
Surrey,
GU2 7YP, England. Please consider the environment before printing this
email. This message should be regarded as confidential. If you have
received
this email in error please notify the sender and destroy it immediately.

Statements of intent shall only become binding when confirmed in hard
copy
by an authorised signatory. The contents of this email may relate to
dealings with other companies within the Detica Limited group of
companies.
Detica Limited is registered in England under No: 1337451. Registered
offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received
this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard
copy by an authorised signatory. The contents of this email may relate
to dealings with other companies within the Detica Limited group of
companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
England.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Please consider the environment before printing this email.

This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.

Mark Diggles wrote:

I have been running it under driver verifier this morning, and have
managed to update symbols, it seems that its crashing on KeSetEvent and
after placing a memory breakpoint this area is null.

I think that its something to with my creating a notification event in
the driver that is waited on in a user app and once this app is
destroyed prematurely something nasty is happening to the kernel event.

This almost certainly has to be your driver overwriting a chunk of
memory somewhere. For example, and I’m just making this up, say you
have a shared KEVENT in a context somewhere, and you clear it and
initialize it before each use, but because of the timing in this odd
circumstance, you get into a condition where you have one callback in
process trying to use the KEVENT while another callback is just starting
and reinitializes the KEVENT.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.