Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTDEV

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Before Posting...

Please check out the Community Guidelines in the Announcements and Administration Category.

My KMDF driver appears to run out of MSI-X interrupts while OEM driver does not.

Erik_DabrowskyErik_Dabrowsky Member - All Emails Posts: 35

I have a server with 6 dual port FibreChannel HBA's in it (12 PCIe functions / device instances).
Each device instance can have up to 32 MSI-X interrupt vectors as defined in the HBA's PCI config space.

When I load the OEM driver for the devices, device manager shows each was assigned 32 IRQs under the resources tab, for a total of 12*32 = 384 MSI-X interrupts among all the instances.

When I load my KMDF driver, 8 of the instances are given 32 vectors, and the remaining 4 are given only 1. This is with no MessageNumberLimit set in the registry.

If I disable 3 of the remaining instances and play around with the MessageNumberLimit on the 9th instance, I find I can get the total number of MSI-X interrupts up to about 273 or so.
The first 8 are assigned 32 and the 9th can have it's MessageNumberLimit set to 17 and it will be assigned 17 IRQs. If I bump it up to 18 (total of 274), I seem to go over some threshold and it only get's assigned 1 MSI-X vector.

If I set the MessageNumberLimit to 16 for each device instance, then all 12 instances are assigned 16 MSI-X vectors.

With all of that said, my question is:
Why can the OEM driver have 32 MSI-X vectors assigned to each of the 12 instances but my KMDF driver apparently cannot?
I assume the OEM driver is an older WDM based driver, but I don't know that for sure.
Is there some sort of limit built in to the KMDF/WDF framework?

Specs:
Server is a SuperMicro with two 8 core Intel Xeon Silver 4215R CPUs.
Motherboard: X11DPX-T
32 GB of RAM
Windows Server 2019.
FibreChannel HBAs are PCIe 3.0
KMDF is 1.19

Thanks for any light you can shed on this discrepancy,
Erik

Comments

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,654

    I assume you’ve compare the MSI settings in the INF files, and made yours identical to those of the other driver?

    Juuust checking…. Cuz that’s really the only thing I know about that can influence this, unless some trickery with resource requirements is involved. Hmmmmm…. Have you looked at the PCIe header on the device, to see what the MSI-x capability says? It’s possible the OEM driver could be “fixing it” before the resources are assigned. I guess…

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Erik_DabrowskyErik_Dabrowsky Member - All Emails Posts: 35

    The OEM inf contains:
    HKR,Interrupt Management,,0x00000010
    HKR, Interrupt Management\MessageSignaledInterruptProperties, 0, 0x00000010
    HKR, Interrupt Management\MessageSignaledInterruptProperties, MSISupported, 0x00010001, 1
    HKR, "Interrupt Management\Affinity Policy", 0, 0x00000010
    HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5

    Mine contains:
    HKR,Interrupt Management,,0x00000010
    HKR,Interrupt Management\MessageSignaledInterruptProperties,,0x00000010
    HKR,Interrupt Management\MessageSignaledInterruptProperties,MSISupported,0x00010001,1

    I can definitely try setting the DevicePolicy and see if that makes any difference.
    I haven't yet had a chance to look at the PCIe header.

    Thanks,
    Erik

  • Erik_DabrowskyErik_Dabrowsky Member - All Emails Posts: 35

    I missed one, the OEM inf also contains:
    HKR, "Interrupt Management\Affinity Policy", GroupPolicy, 0x00010001, 1

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. Sign in or register to get started.

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Internals & Software Drivers 15 November 2021 Live, Online
Writing WDF Drivers TBD Live, Online
Developing Minifilters 7 February 2022 Live, Online
Kernel Debugging 21 March 2022 Live, Online