SRB_FUNCTION_FLUSH being called at DISPATCH_LEVEL

I have a virtual storport miniport driver acting as a RAID driver backed by physical disks. I have added support for using the physical disk caches by adding CachesData = TRUE in the VIRTUAL_HW_INITIALIZATION_DATA struct.

When I receive the SRB_FUNCTION_FLUSH call, I send a synchronous IRP_MJ_FLUSH_BUFFERS call to the individual disks. It worked great for me, but after a rebuild, I am now getting SRB_FUNCTION_FLUSH calls at DISPATCH_LEVEL. Now my driver is violating the Driver Verifier rule about forwarding IRP_MJ_FLUSH_BUFFERS at DISPATCH_LEVEL.

I replaced IRP_MJ_FLUSH_BUFFERS with IRP_MJ_SCSI and sent SRB_FUNCTION_FLUSH to individual disks but now this violates the rule about KeWaitForSingleEvent can’t wait forever at DISPATCH_LEVEL.

If I understand SRB_FUNCTION_FLUSH correctly, I am supposed to return only after the flush is complete (like disk.sys does).

How do I go about resolving this?

The analyze -v and stack trace are for when I call IoCallDriver sending a IRP_MJ_FLUSH_BUFFERS to the TargetDevice.

!analyze -v:

DRIVER_VERIFIER_IOMANAGER_VIOLATION (c9)
The IO manager has caught a misbehaving driver.
Arguments:
Arg1: 0000000000000301, A driver has forwarded an IRP at IRQL > DISPATCH_LEVEL.
Arg2: fffff80569513ba6, The address in the driver's code where the error was detected.
Arg3: ffffdb86e3a5adc0, IRP address.
Arg4: 0000000000000002, Incorrect IRQL value.

Stack trace:

0: kd> k  

 # Child-SP          RetAddr               Call Site  
00 ffffc704`1dc129d0 fffff805`67fdb630     nt!ViErrorFinishReport+0x8e  
01 ffffc704`1dc12a30 fffff805`67febd37     nt!ViGenericVerifyIrpStackDownward+0x110  
02 ffffc704`1dc12b30 fffff805`67fe4082     nt!VfMajorVerifyIrpStackDownward+0xb3  
03 ffffc704`1dc12b90 fffff805`67fe518e     nt!IovpCallDriver1+0x456  
04 ffffc704`1dc12c40 fffff805`67fd8ed6     nt!VfBeforeCallDriver+0x136  
05 ffffc704`1dc12c70 fffff805`67a45da9     nt!IovCallDriver+0x242
06 ffffc704`1dc12cb0 fffff805`69513ba6     nt!IofCallDriver+0x1dbea9
07 ffffc704`1dc12cf0 fffff805`69513ee0     partmgr!PmIo+0xe6
08 ffffc704`1dc12d70 fffff805`679844f7     partmgr!PmGlobalDispatch+0x20
09 ffffc704`1dc12da0 fffff805`67fd8efa     nt!IopfCallDriver+0x53
0a ffffc704`1dc12de0 fffff805`67a45da9     nt!IovCallDriver+0x266
0b ffffc704`1dc12e20 fffff805`6a768dc4     nt!IofCallDriver+0x1dbea9
0c ffffc704`1dc12e60 fffff805`699f351a     MyDiskFilterDriver!MyDriver_Actual_Flush+0x134
0d ffffc704`1dc12f00 fffff805`699f3036     MyDiskFilterDriver!MyDriver_Dispatch_Flush+0x8a
0e ffffc704`1dc12f50 fffff805`69a05e36     MyDiskFilterDriver!MyDriver_Dispatch_Any+0xc6 
0f ffffc704`1dc12f90 fffff805`69a05242     storport!RaidAdapterPostScatterGatherExecute+0x216
10 ffffc704`1dc13020 fffff805`69a0ac21     storport!RaUnitStartIo+0x312
11 ffffc704`1dc13120 fffff805`69a0a200     storport!RaidStartIoPacket+0x4f1
12 ffffc704`1dc13250 fffff805`69a0a56d     storport!RaidUnitSubmitRequest+0x94
13 ffffc704`1dc132a0 fffff805`69a0a28a     storport!RaUnitScsiIrp+0x29d
14 ffffc704`1dc13340 fffff805`679844f7     storport!RaDriverScsiIrp+0x5a
15 ffffc704`1dc13380 fffff805`67fd8efa     nt!IopfCallDriver+0x53
16 ffffc704`1dc133c0 fffff805`67a45da9     nt!IovCallDriver+0x266
17 ffffc704`1dc13400 fffff805`6a8e14be     nt!IofCallDriver+0x1dbea9
18 ffffc704`1dc13440 fffff805`6a8e1192     disk!DiskFlushDispatch+0x17e
19 ffffc704`1dc134a0 fffff805`6a905029     disk!DiskShutdownFlush+0xd2
1a ffffc704`1dc13510 fffff805`6a9083e4     CLASSPNP!ClassShutdownFlush+0x69
1b ffffc704`1dc13540 fffff805`679844f7     CLASSPNP!ClassGlobalDispatch+0x24
1c ffffc704`1dc13570 fffff805`67fd8efa     nt!IopfCallDriver+0x53
1d ffffc704`1dc135b0 fffff805`67a45da9     nt!IovCallDriver+0x266
1e ffffc704`1dc135f0 fffff805`69513ba6     nt!IofCallDriver+0x1dbea9
1f ffffc704`1dc13630 fffff805`69513ee0     partmgr!PmIo+0xe6
20 ffffc704`1dc136b0 fffff805`679844f7     partmgr!PmGlobalDispatch+0x20
21 ffffc704`1dc136e0 fffff805`67fd8efa     nt!IopfCallDriver+0x53
22 ffffc704`1dc13720 fffff805`67a45da9     nt!IovCallDriver+0x266
23 ffffc704`1dc13760 fffff805`6a737b77     nt!IofCallDriver+0x1dbea9
24 ffffc704`1dc137a0 fffff805`6a738794     MyMiniportDriver!MyMinportDriver_Actual_Dispatch+0xc7
25 ffffc704`1dc13800 fffff805`6a7386d3     MyMiniportDriver!MyMiniportDriver_DispatchAny_Internal+0x84 
26 ffffc704`1dc13840 fffff805`6a737637     MyMiniportDriver!MyMiniportDriver_DispatchAny_Internal+0x23
27 ffffc704`1dc13870 fffff805`679844f7     MyMiniportDriver!MyMiniportDriver_DispatchAny+0x57
28 ffffc704`1dc138c0 fffff805`67fd8efa     nt!IopfCallDriver+0x53
29 ffffc704`1dc13900 fffff805`67a45da9     nt!IovCallDriver+0x266
2a ffffc704`1dc13940 fffff805`69601379     nt!IofCallDriver+0x1dbea9
2b ffffc704`1dc13980 fffff805`69be644a     volmgr!VmFlushBuffers+0x129
2c ffffc704`1dc139c0 fffff805`69be01af     Ntfs!NtfsFlushDiskCache+0xca
2d ffffc704`1dc13a20 fffff805`69bdf2b5     Ntfs!LfsFlushLfcb+0xedf
2e ffffc704`1dc13c40 fffff805`6796b518     Ntfs!LfsFlushLfcbCallout+0x25
2f ffffc704`1dc13c70 fffff805`6796b48d     nt!KeExpandKernelStackAndCalloutInternal+0x78
30 ffffc704`1dc13ce0 fffff805`69bcac8b     nt!KeExpandKernelStackAndCalloutEx+0x1d
31 ffffc704`1dc13d20 fffff805`69cbf822     Ntfs!LfsFlushLfcbOnNewStack+0x53
32 ffffc704`1dc13d80 fffff805`69d102dd     Ntfs!LfsFlushToLsnPriv+0x14a
33 ffffc704`1dc13e20 fffff805`69d0f6b9     Ntfs!LfsWriteLfsRestart+0x14d
34 ffffc704`1dc13e80 fffff805`69ccb2bd     Ntfs!LfsWriteRestartArea+0x1b5
35 ffffc704`1dc13fe0 fffff805`69d55cde     Ntfs!NtfsCheckpointVolume+0x102d
36 ffffc704`1dc144e0 fffff805`69d12cc3     Ntfs!NtfsMountVolume+0x1b4e
37 ffffc704`1dc14950 fffff805`69be5924     Ntfs!NtfsCommonFileSystemControl+0xcf
38 ffffc704`1dc14a20 fffff805`6783c975     Ntfs!NtfsFspDispatch+0x654
39 ffffc704`1dc14b70 fffff805`6792ee85     nt!ExpWorkerThread+0x105
3a ffffc704`1dc14c10 fffff805`67a14498     nt!PspSystemThreadStartup+0x55
3b ffffc704`1dc14c60 00000000`00000000     nt!KiStartSystemThread+0x28

Hmmmm… I’m defintely not the StorPort expert here at OSR, but why would you want to complete this SRB synchronously? Can’t you pend it and completing it later, when the flush is done (and the other requirements for SRB_FUNCTION_FLUSH have been met)? Then DISK.SYS will take care of the rest, will it not?

On getting a SRB_FUNCTION_FLUSH, disk.sys constructs a SRB with SCSIOP_SYNCHRONIZE_CACHE and calls ClassPnpSendSrbSynchronous to send a synchronous call to the disk. I planned to do the same.

In classpnp/disk code, the function ClassSendSrbSynchronous waits infinitely with a KeWaitForSingleObject. I am not sure why disk.sys doesn’t trigger the “Can’t wait indefinitely at DISPATCH_LEVEL” Driver Verifier violation for this, like it did in my case.

In classpnp/disk code, the function ClassSendSrbSynchronous waits infinitely

That’s my POINT, dude. Disk waits synchronously, so you don’t HAVE to. Right?

When I receive a SRB_FUNCTION_FLUSH on my virtual RAID disk, I can’t forward it directly to the physical disks. I have to construct fresh IRPs with a) IRP_MJ_FLUSH_BUFFERS or b) IRP_MJ_SCSI and SCSIOP_SYNCHRONIZE_CACHE and send it down and do a KeWaitForSingleEvent until their completion routines fire.

Since I can’t do either at DISPATCH_LEVEL, is there any work around?

Thanks for the replies, Peter.

Either we’re not communicating well, or my lack of StorPort knowledge is so significant that I’m talking nonsense and you’ve so far been too polite to point that out.

Despite the fact that I am not a StorPort expert, let me try one more time to describe what I’ve been suggesting since the start:

I am not aware of any requirement for you to complete the SRBs you receive WITHIN your HwStorStartIo callback. So… send the flush request to a worker thread (which will run at IRQL PASSIVE_LEVEL, by definition) and in that worker thread do what you need to do and (in that same worker thread) call StorPortNotification to complete the request). In your HwStorStartIo return TRUE (thereby indicating that you managed to START the request).

Why is that not do-able?

Heh… It’s definitely my misunderstanding of what you were trying to point out. I was looking for a reason as to why the same code (more or less) in disk.sys does not violate Driver Verifier but mine does. (Reasons include: I already coded it, and it appeared to work before, and the stack trace shows no reason for the thread to be at DISPATCH_LEVEL).

It does absolutely make sense to do the way you have suggested and I actually started down that path today morning. So thank you for that.

Cheers.

PS: I have been reading your replies for more than a decade. Huge fan!