My KMDF driver appears to run out of MSI-X interrupts while OEM driver does not.

NTDEV
1

I have a server with 6 dual port FibreChannel HBA’s in it (12 PCIe functions / device instances).
Each device instance can have up to 32 MSI-X interrupt vectors as defined in the HBA’s PCI config space.

When I load the OEM driver for the devices, device manager shows each was assigned 32 IRQs under the resources tab, for a total of 12*32 = 384 MSI-X interrupts among all the instances.

When I load my KMDF driver, 8 of the instances are given 32 vectors, and the remaining 4 are given only 1. This is with no MessageNumberLimit set in the registry.

If I disable 3 of the remaining instances and play around with the MessageNumberLimit on the 9th instance, I find I can get the total number of MSI-X interrupts up to about 273 or so.
The first 8 are assigned 32 and the 9th can have it’s MessageNumberLimit set to 17 and it will be assigned 17 IRQs. If I bump it up to 18 (total of 274), I seem to go over some threshold and it only get’s assigned 1 MSI-X vector.

If I set the MessageNumberLimit to 16 for each device instance, then all 12 instances are assigned 16 MSI-X vectors.

With all of that said, my question is:
Why can the OEM driver have 32 MSI-X vectors assigned to each of the 12 instances but my KMDF driver apparently cannot?
I assume the OEM driver is an older WDM based driver, but I don’t know that for sure.
Is there some sort of limit built in to the KMDF/WDF framework?

Specs:
Server is a SuperMicro with two 8 core Intel Xeon Silver 4215R CPUs.
Motherboard: X11DPX-T
32 GB of RAM
Windows Server 2019.
FibreChannel HBAs are PCIe 3.0
KMDF is 1.19

Thanks for any light you can shed on this discrepancy,
Erik

2

I assume you’ve compare the MSI settings in the INF files, and made yours identical to those of the other driver?

Juuust checking…. Cuz that’s really the only thing I know about that can influence this, unless some trickery with resource requirements is involved. Hmmmmm…. Have you looked at the PCIe header on the device, to see what the MSI-x capability says? It’s possible the OEM driver could be “fixing it” before the resources are assigned. I guess…

Peter

3

The OEM inf contains:
HKR,Interrupt Management,0x00000010
HKR, Interrupt Management\MessageSignaledInterruptProperties, 0, 0x00000010
HKR, Interrupt Management\MessageSignaledInterruptProperties, MSISupported, 0x00010001, 1
HKR, “Interrupt Management\Affinity Policy”, 0, 0x00000010
HKR, “Interrupt Management\Affinity Policy”, DevicePolicy, 0x00010001, 5

Mine contains:
HKR,Interrupt Management,0x00000010
HKR,Interrupt Management\MessageSignaledInterruptProperties,0x00000010
HKR,Interrupt Management\MessageSignaledInterruptProperties,MSISupported,0x00010001,1

I can definitely try setting the DevicePolicy and see if that makes any difference.
I haven’t yet had a chance to look at the PCIe header.

Thanks,
Erik

4

I missed one, the OEM inf also contains:
HKR, “Interrupt Management\Affinity Policy”, GroupPolicy, 0x00010001, 1