Bug code DRIVER_VERIFIER_IOMANAGER_VIOLATION; 0x302

Hi all,
Here is my dump analysis, and I wonder what is wrong with my system.
Thanks & regards.

2: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_VERIFIER_IOMANAGER_VIOLATION (c9)
The IO manager has caught a misbehaving driver.
Arguments:
Arg1: 00000302, Code that specifies the violation
Arg2: 84a992a9
Arg3: 84448ed8
Arg4: 00000001

Debugging Details:

Page 6a548 not present in the dump file. Type “.hh dbgerr004” for details
PEB is paged out (Peb.Ldr = 7ffd900c). Type “.hh dbgerr001” for details
PEB is paged out (Peb.Ldr = 7ffd900c). Type “.hh dbgerr001” for details

BUGCHECK_STR: 0xc9_302

DRIVER_VERIFIER_IO_VIOLATION_TYPE: 302

FAULTING_IP:
nt!IopParseDevice+ed7
84a992a9 8945e0 mov dword ptr [ebp-20h],eax

FOLLOWUP_IP:
cvfsfilter+1027
8b188027 8945cc mov dword ptr [ebp-34h],eax

IRP_ADDRESS: 84448ed8

DEVICE_OBJECT: 00000000

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

PROCESS_NAME: fsmpm.exe

CURRENT_IRQL: 2

LAST_CONTROL_TRANSFER: from 84b88f03 to 84930f20

STACK_TEXT:
904cf4bc 84b88f03 000000c9 00000302 84a992a9 nt!KeBugCheckEx+0x1e
904cf4dc 84b8b2cd 84a992a9 904cf518 00000001 nt!VerifierBugCheckIfAppropriate+0x30
904cf4f4 84b8b63b 00000302 84a992a9 00000001 nt!ViErrorFinishReport+0xc9
904cf560 84b92849 00000302 84a992a9 00000001 nt!VfErrorReport11+0x58
904cf57c 84b8b076 84448fd8 90d41b00 84a992a9 nt!ViGenericVerifyNewIrp+0x5f
904cf590 84b893e1 92e794a8 84448ed8 90d41b00 nt!VfMajorVerifyNewIrp+0x52
904cf5f4 84b88d33 92e630d0 84448ed8 904cf624 nt!IovpCallDriver1+0x423
904cf604 84b83670 8f31d670 92e98998 8f31d670 nt!VfBeforeCallDriver+0xe7
904cf624 8488954a 84448fd8 92e989f4 8f31d670 nt!IovCallDriver+0x206
904cf638 84a992a9 bba4673c 904cf7e0 00000000 nt!IofCallDriver+0x1b
904cf710 84a78ac5 8f31d988 8c7a96e0 8d79f8c0 nt!IopParseDevice+0xed7
904cf78c 84a88ed6 00000000 904cf7e0 00000240 nt!ObpLookupObjectName+0x4fa
904cf7ec 84a7f9b4 904cf978 887a96e0 00000000 nt!ObOpenObjectByName+0x165
904cf868 84a85b3a 904cf968 00000000 904cf978 nt!IopCreateFile+0x673
904cf8b0 848901ea 904cf968 00000000 904cf978 nt!NtOpenFile+0x2a
904cf8b0 8488e561 904cf968 00000000 904cf978 nt!KiFastCallEntry+0x12a
904cf940 84ab975e 904cf968 00000000 904cf978 nt!ZwOpenFile+0x11
904cf994 8b188027 904cf9cc 00000000 904cf9c8 nt!IoGetDeviceObjectPointer+0x59
WARNING: Stack unwind information not available. Following frames may be wrong.
904cf9e0 8b1880a7 92e9bb2c 904cfa00 904cfa08 cvfsfilter+0x1027
904cfa10 8b187b14 9c540b58 89e80dc0 92e9bb2c cvfsfilter+0x10a7
904cfa88 8b188aad 9c540b58 89e80dc0 13882834 cvfsfilter+0xb14
904cfab8 8b18898a 89e80dc0 8454af20 00000002 cvfsfilter+0x1aad
904cfae0 84b836c3 89e80dc0 8454af20 8454afd8 cvfsfilter+0x198a
904cfb04 8488954a 00000000 8454affc 89e80dc0 nt!IovCallDriver+0x258
904cfb18 849edb35 92e86d58 8454af20 8454af20 nt!IofCallDriver+0x1b
904cfb30 849ed1ef 92e86d58 8454af20 bba46b54 nt!RawReadWriteDeviceControl+0x14b
904cfb78 84b836c3 92e86ca0 8454af20 9c5a4920 nt!RawDispatch+0x20a
904cfb9c 8488954a 00000000 8454af20 92e86ca0 nt!IovCallDriver+0x258
904cfbb0 879d23e8 8454af20 9c5a4920 92e8b390 nt!IofCallDriver+0x1b
904cfbdc 84b836c3 9c5a4920 8454af20 9c59dd88 fltmgr!FltpDispatch+0xe2
904cfc00 8488954a 00000000 8454af20 9c5a4920 nt!IovCallDriver+0x258
904cfc14 84a7d99f 9c59dd88 8454af20 8454afd8 nt!IofCallDriver+0x1b
904cfc34 84a80b71 9c5a4920 9c59dd88 00000000 nt!IopSynchronousServiceTail+0x1f8
904cfcd0 84ac73f4 9c5a4920 8454af20 00000000 nt!IopXxxControlFile+0x6aa
904cfd04 848901ea 000002bc 00000000 00000000 nt!NtDeviceIoControlFile+0x2a
904cfd04 774170b4 000002bc 00000000 00000000 nt!KiFastCallEntry+0x12a
00d0f234 00000000 00000000 00000000 00000000 0x774170b4

STACK_COMMAND: kb

SYMBOL_STACK_INDEX: 12

SYMBOL_NAME: cvfsfilter+1027

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: cvfsfilter

IMAGE_NAME: cvfsfilter.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4cffc080

FAILURE_BUCKET_ID: 0xc9_302_VRF_cvfsfilter+1027

BUCKET_ID: 0xc9_302_VRF_cvfsfilter+1027

Followup: MachineOwner

2: kd> !irp 84448ed8
Irp is active with 5 stacks 6 is current (= 0x84448ffc)
No Mdl: No System Buffer: Thread 9c5d1230: Irp is completed.
cmd flg cl Device File Completion-Context
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 0 00000000 92e98998 00000000-00000000

Args: 904cf65c 01000040 00000000 00000000
2: kd> !fileobj 92e98998

Device Object: 0x8f31d988 \Driver\Disk
Vpb is NULL

Flags: 0x800
Direct Device Open

CurrentByteOffset: 0

2: kd> !devstack 0x8f31d988
!DevObj !DrvObj !DevExt ObjectName
8f31d670 \Driver\partmgr 8f31d728

8f31d988 \Driver\Disk 8f31da40
8f31dee0 \DRIVER\VERIFIER_FILTER8f31df98
8d7b7030 \Driver\Upbase 8d7b70e8
!DevNode 8d7d7300 :
DeviceInst is “UPIO\DiskHUAWEI__S5300___________________________1___.001\1&146a97f6&2a&3639383231393231303038333736323130326461626532373030303030303033”
ServiceName is “disk”
2: kd> dt 904cf978 _OBJECT_ATTRIBUTES
hal!_OBJECT_ATTRIBUTES
+0x000 Length : 0x18
+0x004 RootDirectory : (null)
+0x008 ObjectName : 0x904cf9cc _UNICODE_STRING “\Device\Harddisk1\DR1”
+0x00c Attributes : 0x240
+0x010 SecurityDescriptor : (null)
+0x014 SecurityQualityOfService : (null)

And the IRQL when the crash happening is 2(DISPATCH_LEVEL).

Show your handler for IRP_MJ_CREATE

switch (pstIrpStack->MajorFunction)
{
case IRP_MJ_CREATE:
case IRP_MJ_CLOSE:
ulStatus = STATUS_SUCCESS;
/*lint -save -e40 -e63 for status*/
pstIrp->IoStatus.Status = ulStatus;
/*lint -restore*/
IoCompleteRequest(pstIrp, IO_NO_INCREMENT);
IoReleaseRemoveLock(&pstVLunDevExt->stDiskPdoRemoveLock, pstIrp);
return ulStatus;


}

My driver is a HBA driver(\Driver\Upbase), and the device “UPIO\DiskHUAWEI__S5300___________________________1___.001\1&146a97f6&2a&36393832
31393231303038333736323130326461626532373030303030303033”
is my device. Is it the reason that cvfsfilter call IoGetDeviceObjectPointer at IRQL=2 which caused the crash? It seems that the IRP is not pass to my driver yet. And there is one more thing I don’t understand, why "WARNING: Stack unwind information not available. Following frames may be wrong.
" is desplayed and what’s the reason for it.

The latest documentation on msdn.com says:

"A driver has forwarded an IRP at IRQL >= APC_LEVEL.

The I/O Manager will need to queue an APC to complete this request. The APC will not be able to run because the caller is already at APC level, so the caller is likely to deadlock. (IRQL value specified)"

Is CVFSFILTER your driver?

No, it’s Quantum’s StorNext. I can know is this software’s service will try to open each disk when it starts. And this crash can always reproduces when I start the service. But if there is no disks enumerated by my driver, it won’t happen.

And this crash only reproduces on Emulex’s adapter.

nt!RawReadWriteDeviceControl+0x14b is running at PASSIVE_LEVEL, but when entering cvfsfilter it seems that the thread is turn to a APC or DPC or something. Am I right?

What kind of driver is yours? Do you ever call KeEnterCriticalSection or KeEnterGuardedRegion, or use fast mutexes? Check whether you ever forget to restore IRQL or undo these functions.

A full-port driver(WDM driver). Fast mutex is used, but which part code is most like cause this problem, so I can find it?

While you hold the fast mutex, don’t call any function that requires PASSIVE_LEVEL. If a function acquires the mutex, it MUST release it before returning. Especially if it’s a dispatch or callback function.

A fast mutex changes IRQL to APC_LEVEL. If you return back to OS without releasing the mutex, you screwed the global IRQL.