Crash IRQL_NOT_LESS_OR_EQUAL in IopCompleteRequest.

Hey we are getting a crash when our whitelisting product and another third party is installed together. THis crash does not come with either of these products alone. WE used driver verifier but even then we are having trouble getting the driver at fault. I understand what the bugcheck is saying but having trouble deciphering the faulty driver.
I ll call my product as product 1 and other product as product2 in the stack.

OS:: Windows 7 Kernel Version 7601 (Service Pack 1) MP (4 procs) Free x64

!address fffff98003b12000
Usage: System Cache
Base Address: fffff98003b00000 End Address: fffff98003b40000
Region Size: 00000000`00040000
VA Type: SystemDynamicSpace
VACB: fffffa800670de80 [$Mft]

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.

Arguments:
Arg1: fffff98003b12000, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, bitfield :
Arg4: fffff800032de756, address which referenced memory

WRITE_ADDRESS: fffff98003b12000
CURRENT_IRQL: 2
nt!KeBugCheckEx
nt!KiBugCheckDispatch+0x69
nt!KiPageFault+0x260
nt!IopCompleteRequest+0xc76
nt!KiDeliverApc+0x1c7
nt!KiCheckForKernelApcDelivery+0x25
nt!MmAccessFault+0x1a3c
nt!KiPageFault+0x16e
nt! ?? ::NNGAKEGL::string'+0x2b2f5 nt!NtMapViewOfSection+0x2bd nt!KiSystemServiceCopyEnd+0x13 ntdll!ZwMapViewOfSection+0xa KERNELBASE!BasepLoadLibraryAsDataFileInternal+0x27d KERNELBASE!LoadLibraryExW+0x1b7 PRODUCT2's dll+0xe86b 0x000007fef8230000
0x2cd780
0x2cd810

IRPS qued to this thread are:
fffff98167822ee0: (0006,0118) Flags: 40060000 Mdl: 00000000
fffff98167828ee0: (0006,0118) Flags: 40060000 Mdl: 00000000
fffff9816858cee0: (0006,0118) Flags: 40060000 Mdl: 00000000
fffff981678e6ee0: (0006,0118) Flags: 40060000 Mdl: 00000000
fffff9816781eb80: (0006,0478) Flags: 40060000 Mdl: 00000000

2: kd> !irp fffff98167822ee0
Irp is active with 1 stacks 3 is current (= 00000000)
No Mdl: No System Buffer: Thread fffff800032ddae0: Irp is completed.
cmd flg cl Device File Completion-Context
[e, 0] 0 10 fffffa800eb917b0 00000000 00000000-00000000
\Driver\PRODUCT2
Args: 00000000 00000000 00000000 fffff88009d22bb0

Similarly for fffff98167828ee0,fffff9816858cee0,fffff981678e6ee0.
Last irp is with NTFS:
2: kd> !irp fffff9816781eb80
Irp is active with 13 stacks 13 is current (= 0xfffff9816781efb0)
No Mdl: No System Buffer: Thread fffffa8011197590: Irp stack trace.
cmd flg cl Device File Completion-Context
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 10 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000

[c, 2] 0 1 fffffa8007abe030 fffffa801119cb80 00000000-00000000 pending
\FileSystem\Ntfs
Args: 00000020 00000011 00000000 00000000

Please help me debug this problem.

IopCompleteRequest is the special kernel mode APC for I/O completion, it
performs the last stage of I/O completion before freeing the IRP. This
routine is tripping over an invalid virtual address while completing an IRP,
so it is reasonable to assume that something is wrong with the IRP being
completed. Also note the invalid virtual address is on a page boundary
(fffff98003b12000), so it’s possible that this is a buffer overflow. Any
chance you’re setting the Information field of the IRP to a size that is
larger than the user request?

Also note that at some point during this routine the IRP is dequeued from
the thread, thus the “bad” IRP may or may not still be queued to the thread.

You really need to post the entire output of !analyze -v, it has lots of
clues that we could use to guide you further. However, the next steps that
you need to follow are:

  1. Find the trap frame in the !analyze output and pass the address to .trap

  2. Look at the surrounding assembly and figure out where the invalid virtual
    address came from

  3. Figure out which IRP is currently being completed with
    IopCompleteRequest, it will be on the stack or in a register somewhere

If you make a dump available somewhere I can try to poke at it and collect
some additional details.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

Hey we are getting a crash when our whitelisting product and another third
party is installed together. THis crash does not come with either of these
products alone. WE used driver verifier but even then we are having trouble
getting the driver at fault. I understand what the bugcheck is saying but
having trouble deciphering the faulty driver.
I ll call my product as product 1 and other product as product2 in the
stack.

OS:: Windows 7 Kernel Version 7601 (Service Pack 1) MP (4 procs) Free x64

!address fffff98003b12000
Usage: System Cache
Base Address: fffff98003b00000 End Address: fffff98003b40000
Region Size: 00000000`00040000
VA Type: SystemDynamicSpace
VACB: fffffa800670de80 [$Mft]

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at
an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.

Arguments:
Arg1: fffff98003b12000, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, bitfield :
Arg4: fffff800032de756, address which referenced memory

WRITE_ADDRESS: fffff98003b12000
CURRENT_IRQL: 2
nt!KeBugCheckEx
nt!KiBugCheckDispatch+0x69
nt!KiPageFault+0x260
nt!IopCompleteRequest+0xc76
nt!KiDeliverApc+0x1c7
nt!KiCheckForKernelApcDelivery+0x25
nt!MmAccessFault+0x1a3c
nt!KiPageFault+0x16e
nt! ?? ::NNGAKEGL::string'+0x2b2f5 nt!NtMapViewOfSection+0x2bd nt!KiSystemServiceCopyEnd+0x13 ntdll!ZwMapViewOfSection+0xa KERNELBASE!BasepLoadLibraryAsDataFileInternal+0x27d KERNELBASE!LoadLibraryExW+0x1b7 PRODUCT2's dll+0xe86b 0x000007fef8230000
0x2cd780
0x2cd810

IRPS qued to this thread are:
fffff98167822ee0: (0006,0118) Flags: 40060000 Mdl: 00000000
fffff98167828ee0: (0006,0118) Flags: 40060000 Mdl: 00000000
fffff9816858cee0: (0006,0118) Flags: 40060000 Mdl: 00000000
fffff981678e6ee0: (0006,0118) Flags: 40060000 Mdl: 00000000
fffff9816781eb80: (0006,0478) Flags: 40060000 Mdl: 00000000

2: kd> !irp fffff98167822ee0
Irp is active with 1 stacks 3 is current (= 00000000)
No Mdl: No System Buffer: Thread fffff800032ddae0: Irp is completed.
cmd flg cl Device File Completion-Context
[e, 0] 0 10 fffffa800eb917b0 00000000 00000000-00000000
\Driver\PRODUCT2
Args: 00000000 00000000 00000000 fffff88009d22bb0

Similarly for fffff98167828ee0,fffff9816858cee0,fffff981678e6ee0.
Last irp is with NTFS:
2: kd> !irp fffff9816781eb80
Irp is active with 13 stacks 13 is current (= 0xfffff9816781efb0)
No Mdl: No System Buffer: Thread fffffa8011197590: Irp stack trace.
cmd flg cl Device File Completion-Context
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 10 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000

[c, 2] 0 1 fffffa8007abe030 fffffa801119cb80 00000000-00000000
pending
\FileSystem\Ntfs
Args: 00000020 00000011 00000000 00000000

Please help me debug this problem.

!analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_VERIFIER_IOMANAGER_VIOLATION (c9)
The IO manager has caught a misbehaving driver.
Arguments:
Arg1: 0000000c, Invalid IOSB in IRP at APC IopCompleteRequest (appears to be on
stack that was unwound)
Arg2: 9ef1ec08, IOSB address
Arg3: a2026f68, IRP address
Arg4: 00000000

Debugging Details:

*** ERROR: Module load completed but symbols could not be loaded for Product2.sys

BUGCHECK_STR: 0xc9_c

DRIVER_VERIFIER_IO_VIOLATION_TYPE: c

IOSB_ADDRESS: ffffffff9ef1ec08

IRP_ADDRESS: 00026f68

DEVICE_OBJECT: 8adeeed8

DRIVER_OBJECT: 8aded4b0

IMAGE_NAME: Product2.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 53710ea9

MODULE_NAME: Product2

FAULTING_MODULE: 90fa0000 Product2

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

PROCESS_NAME: SearchProtocol

CURRENT_IRQL: 2

LAST_CONTROL_TRANSFER: from 830f9083 to 83095110

STACK_TEXT:
9ef1f5dc 830f9083 00000003 47960f61 00000065 nt!RtlpBreakWithStatusInstruction
9ef1f62c 830f9b81 00000003 00000000 a2026f68 nt!KiBugCheckDebugBreak+0x1c
9ef1f9f0 830f8f20 000000c9 0000000c 9ef1ec08 nt!KeBugCheck2+0x68b
9ef1fa10 8334b9f1 000000c9 0000000c 9ef1ec08 nt!KeBugCheckEx+0x1e
9ef1fa30 8309b095 47960339 9e53d890 a2026fa8 nt!IovpCompleteRequest+0x3c
9ef1fa74 830cb746 00026fa8 9ef1faa0 9ef1faac nt!IopCompleteRequest+0x4a
9ef1fac4 830424bd 00000000 00000000 00000000 nt!KiDeliverApc+0x111
9ef1fad8 83261def 00000000 00000000 98813120 nt!KiCheckForKernelApcDelivery+0x24
9ef1fb24 83261e3a 9e566a58 9ef1fbb0 00000000 nt!MiMapViewOfSection+0x2d3
9ef1fb54 83262599 98813120 9e566a58 9ef1fbb0 nt!MmMapViewOfSection+0x2a
9ef1fbd0 860d7663 000003bc ffffffff 00725110 nt!NtMapViewOfSection+0x204
9ef1fc04 830581ea 000003bc ffffffff 00725110 Product1+0x63 [e:\code\trunk\src\solidifier\casp\casp_hooks.c @ 411]
9ef1fc04 778d70b4 000003bc ffffffff 00725110 nt!KiFastCallEntry+0x12a
00725024 778d5c34 778f067d 000003bc ffffffff ntdll!KiFastSystemCallRet
00725028 778f067d 000003bc ffffffff 00725110 ntdll!NtMapViewOfSection+0xc
0072507c 778f075a 000003bc 00000000 00000000 ntdll!LdrpMapViewOfSection+0xc7
00725114 778f4d73 00725170 00725660 00000000 ntdll!LdrpFindOrMapDll+0x303
00725384 778f53e4 0015f248 00725660 755f02d2 ntdll!LdrpLoadImportModule+0x287
0072540c 778f54ee 00725660 0015f248 0072544c ntdll!LdrpHandleOneNewFormatImportDescriptor+0x78
0072543c 778f5470 00725660 7560ac28 755f0290 ntdll!LdrpHandleNewFormatImportDescriptors+0x96
007254c4 778f0461 0015f248 00725660 77e5bf7a ntdll!LdrpProcessStaticImports+0x22a
00725634 778f232c 00725694 00725660 00000000 ntdll!LdrpLoadDll+0x314
00725668 75b788ee 001626c4 007256a8 00725694 ntdll!LdrLoadDll+0x92
007256a0 76b73c12 00000000 00000000 00000001 KERNELBASE!LoadLibraryExW+0x15a
007256b4 76d928ac 0016f370 00000000 0058c78c kernel32!LoadLibraryW+0x11
007256d4 76d8b416 00729a44 70b72f25 000003b0 ADVAPI32!AccProvpLoadMartaFunctions+0x68
007256dc 70b72f25 000003b0 00000001 00000007 ADVAPI32!GetSecurityInfo+0x35
00729a44 70b72b2e 0072aa20 00000000 00000000 mssph!CFileAccessor::Init+0x596
00729a88 70b72ab8 0072aa20 00000000 00000000 mssph!CFileHandler::CreateAccessorInternal+0x89
00729aa4 00675004 0058c2a8 0072aa20 00000000 mssph!CFileHandler::CreateAccessorEx+0x2b
00729b38 00675118 0072aa20 00000000 00674260 SearchProtocolHost!CProtocolHandlers::GetURLAccessor+0x238
00729ba0 0067252f 0058c010 00729d10 00000000 SearchProtocolHost!GetUrlAccessor+0x98
0072f7b4 76b73c45 00117fb8 0072f800 778f37f5 SearchProtocolHost!CFilterThread::Thread+0x617
0072f7c0 778f37f5 00117fb8 77e5114e 00000000 kernel32!BaseThreadInitThunk+0xe
0072f800 778f37c8 00672184 00117fb8 00000000 ntdll!__RtlUserThreadStart+0x70
0072f818 00000000 00672184 00117fb8 00000000 ntdll!_RtlUserThreadStart+0x1b

STACK_COMMAND: kb

FOLLOWUP_NAME: MachineOwner

FAILURE_BUCKET_ID: 0xc9_c_VRF_IMAGE_Product2.sys

BUCKET_ID: 0xc9_c_VRF_IMAGE_Product2.sys

Followup: MachineOwner

kd> !thread
THREAD 9e53d890 Cid 0a68.0ad8 Teb: 7ffdc000 Win32Thread: fe9c3dd8 RUNNING on processor 0
IRP List:
a2026f68: (0006,0094) Flags: 40020000 Mdl: 00000000
Not impersonating
DeviceMap 87408008
Owning Process 9e566a58 Image: SearchProtocolHost.exe
Attached Process N/A Image: N/A
Wait Start TickCount 8094 Ticks: 1 (0:00:00:00.015)
Context Switch Count 37 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.031
Win32 Start Address SearchProtocolHost!CFilterThread::Thread (0x00672184)
Stack Init 9ef1fed0 Current 9ef1f610 Base 9ef20000 Limit 9ef1d000 Call 0
Priority 15 BasePriority 4 UnusualBoost 10 ForegroundBoost 0 IoPriority 0 PagePriority 1
ChildEBP RetAddr Args to Child
9ef1f5dc 830f9083 00000003 47960f61 00000065 nt!RtlpBreakWithStatusInstruction (FPO: [1,0,0])
9ef1f62c 830f9b81 00000003 00000000 a2026f68 nt!KiBugCheckDebugBreak+0x1c
9ef1f9f0 830f8f20 000000c9 0000000c 9ef1ec08 nt!KeBugCheck2+0x68b
9ef1fa10 8334b9f1 000000c9 0000000c 9ef1ec08 nt!KeBugCheckEx+0x1e
9ef1fa30 8309b095 47960339 9e53d890 a2026fa8 nt!IovpCompleteRequest+0x3c
9ef1fa74 830cb746 00026fa8 9ef1faa0 9ef1faac nt!IopCompleteRequest+0x4a
9ef1fac4 830424bd 00000000 00000000 00000000 nt!KiDeliverApc+0x111
9ef1fad8 83261def 00000000 00000000 98813120 nt!KiCheckForKernelApcDelivery+0x24
9ef1fb24 83261e3a 9e566a58 9ef1fbb0 00000000 nt!MiMapViewOfSection+0x2d3
9ef1fb54 83262599 98813120 9e566a58 9ef1fbb0 nt!MmMapViewOfSection+0x2a
9ef1fbd0 860d7663 000003bc ffffffff 00725110 nt!NtMapViewOfSection+0x204
9ef1fc04 830581ea 000003bc ffffffff 00725110 Product1!casp_hook_map_view_of_section+0x63 (FPO: [Non-Fpo]) (CONV: stdcall) [e:\code\trunk\src\solidifier\casp\casp_hooks.c @ 411]
9ef1fc04 778d70b4 000003bc ffffffff 00725110 nt!KiFastCallEntry+0x12a (FPO: [0,3] TrapFrame @ 9ef1fc34)
00725024 778d5c34 778f067d 000003bc ffffffff ntdll!KiFastSystemCallRet (FPO: [0,0,0])
00725028 778f067d 000003bc ffffffff 00725110 ntdll!NtMapViewOfSection+0xc (FPO: [10,0,0])
0072507c 778f075a 000003bc 00000000 00000000 ntdll!LdrpMapViewOfSection+0xc7 (FPO: [Non-Fpo])
00725114 778f4d73 00725170 00725660 00000000 ntdll!LdrpFindOrMapDll+0x303 (FPO: [Non-Fpo])
00725384 778f53e4 0015f248 00725660 755f02d2 ntdll!LdrpLoadImportModule+0x287 (FPO: [Non-Fpo])
0072540c 778f54ee 00725660 0015f248 0072544c ntdll!LdrpHandleOneNewFormatImportDescriptor+0x78 (FPO: [Non-Fpo])
0072543c 778f5470 00725660 7560ac28 755f0290 ntdll!LdrpHandleNewFormatImportDescriptors+0x96 (FPO: [Non-Fpo])
007254c4 778f0461 0015f248 00725660 77e5bf7a ntdll!LdrpProcessStaticImports+0x22a (FPO: [Non-Fpo])
00725634 778f232c 00725694 00725660 00000000 ntdll!LdrpLoadDll+0x314 (FPO: [Non-Fpo])
00725668 75b788ee 001626c4 007256a8 00725694 ntdll!LdrLoadDll+0x92 (FPO: [Non-Fpo])
007256a0 76b73c12 00000000 00000000 00000001 KERNELBASE!LoadLibraryExW+0x15a (FPO: [Non-Fpo])
007256b4 76d928ac 0016f370 00000000 0058c78c kernel32!LoadLibraryW+0x11 (FPO: [Non-Fpo])
007256d4 76d8b416 00729a44 70b72f25 000003b0 ADVAPI32!AccProvpLoadMartaFunctions+0x68 (FPO: [Non-Fpo])
007256dc 70b72f25 000003b0 00000001 00000007 ADVAPI32!GetSecurityInfo+0x35 (FPO: [Non-Fpo])
00729a44 70b72b2e 0072aa20 00000000 00000000 mssph!CFileAccessor::Init+0x596 (FPO: [Non-Fpo])
00729a88 70b72ab8 0072aa20 00000000 00000000 mssph!CFileHandler::CreateAccessorInternal+0x89 (FPO: [Non-Fpo])
00729aa4 00675004 0058c2a8 0072aa20 00000000 mssph!CFileHandler::CreateAccessorEx+0x2b (FPO: [Non-Fpo])
00729b38 00675118 0072aa20 00000000 00674260 SearchProtocolHost!CProtocolHandlers::GetURLAccessor+0x238 (FPO: [Non-Fpo])
00729ba0 0067252f 0058c010 00729d10 00000000 SearchProtocolHost!GetUrlAccessor+0x98 (FPO: [Non-Fpo])
0072f7b4 76b73c45 00117fb8 0072f800 778f37f5 SearchProtocolHost!CFilterThread::Thread+0x617 (FPO: [Non-Fpo])
0072f7c0 778f37f5 00117fb8 77e5114e 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0072f800 778f37c8 00672184 00117fb8 00000000 ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0072f818 00000000 00672184 00117fb8 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

kd> !irp a2026f68
Irp is active with 1 stacks 3 is current (= 00000000)
No Mdl: No System Buffer: Thread 9e53d8d0: Irp is completed.
cmd flg cl Device File Completion-Context
[e, 0] 0 10 8adeeed8 00000000 00000000-00000000
\Driver\Product2
Args: 00000000 00000000 00000000 9ef1ec04
THis irp is also shown queued to the thread.

Access violation is coming at 9ef1ec04+4 and it belongs to address range of the stack of this thread. Since the IRP logs with verifier show that IRP was allocated by Product2 and even completed by it I have involved them in debugging this. Let me know if any pointers you get from this. Thanks a lot for you reply and guidance.