Setting Up MSIX Interrupts For a PCI Device [WDF]

NTDEV
1

Hi Guys,
This is a general question regarding WDF drivers written for a PCIe device which has multiple interrupt sources (for example multiple queues). In my interrupt service routine I would like to identify every source without reading the “interrupt status” register. The message code that we get in the ISR should give me the source of the interrupt, so that I do not need to go and read a status register from the hardware. Few questions:

  1. Can someone point me to the sample source code which does this?
  2. Any pointers to WDF documentation about this.
  3. What happens if there are two interrupts (hence two messages) at the same time? I think there will two interrupts with two messages. but please correct me if I am missing something here.

Any help highly appreciated.

Thanks
Ajitabh

2

Why don’t you want to read the “interrupt status” register? That’s what it’s for, for gosh sakes.

In general, all of the interrupts for a single device are routed to the same CPU, so multiple MSI interrupts should be serialized. I doubt there is any guarantee of that.

1 Like
3

Why don’t you want to read the “interrupt status” register

Because it’s a serializing operation. One of the goals of MSI/MSI-X is to make it so that you don’t need to touch ANY device registers in your ISR (or, in many cases, any time after device setup/initialization). This avoids the slow-downs due to serialization. In high -speed devices/transfers, like the OP, it’s usually one of my goals to not touch any device registers in the ISR. You use the MSI message number (for example) to tell you which set of memory resident data structures you need to process in your ISR (and/or DPC).

all of the interrupts for a single device are routed to the same CPU

Hmmmm… really? I’ve never heard this in all the years I’ve been writing drivers… and I’m not sure it matches my experience. Windows definitely connects devices to all CPUs… and I was under the impression that the ICH/PCH had “magic” to determine which CPU to which to route the interrupt from a given device (it’s in one of the “colored” Intel books, IIRC).

It’s definitely not the case for MSI-X where the device can actually CHOOSE the CPU to which to target the interrupt.

The message code that we get in the ISR should give me the source of the interrupt, so that I do not need to go and read a status register from the hardware

Well, yes or no, depending. The prototype for your ISR is:

BOOLEAN 
DeviceInterrupt_EvtInterruptIsr(
    _In_ WDFINTERRUPT Interrupt,
    _In_ ULONG MessageID
    )

Right? So, for MSI type interrupts, you just look at the MessageID in your ISR to determine which interrupt is being services. You’ll get one Interrupt resource (passed to you in EvtDevicePrepareHardware) and you need to create one WDFINTERRUPT for each message thats assigned to you for that single Interrupt resource (see the u.MessageInterrupt.Raw.MessageCount field of the raw resource descriptor). The ISRs that you connect to this (one) interrupt resource will be called for all your MSIs. You differentiate using the MessageID field.

Except… you specifically said MSI-X in your title, not MSI.

For MSI-X things are slightly different. You’ll get one interrupt resource (passed to you in EvtDevicePrepareHardware) for each MSI-X interrupt that you’re granted. Therefore the ISR that’s called will imply which interrupt was generated. You will need to connect these interrupts (calling WdfInterruptCreate) from your EvtDevicePrepareHardware Event Processing Callback.

The sort of basic documentation for this is here… it could definitely be more helpful, as it leaves a lot of pragmatic things unaddressed. But my comments above should fill those missing things in sufficiently for you.

Peter

ETA: Clarify that you need to call WdfInterruptCreate for each MSI, which was not clear in the original post

4

Thank you peter. This is really helpful. I will go through the documentation on this and will also look at samples. If you can recommend any samples that will be great.

5

I’m not sure why you’re looking for a sample. There’s really nothing to see, particularly in the ISR.

The only even slightly tricky item is calling WdfInterruptCreate from your EvtDevicePrepareHardware, which is badly described in the docs, but which might want to look something like this:

            //
            // We expect MSI-X resources only.  The device does not fall-back to an LBI. 
            // NOTE: THE CODE BELOW DOES NOT WORK FOR MSI
            //
            case CmResourceTypeInterrupt: {

                MyTracePrint(INFO,"Resource %lu: Interrupt\n", i);

                if(totalInterruptsFound < MY_DEVICE_EXPECTED_NUMBER_OF_INTERRUPTS) {

                    resourceRaw = WdfCmResourceListGetDescriptor(Resources, i);

                    if(resourceTrans->Flags & CM_RESOURCE_INTERRUPT_MESSAGE) {

                        WDF_INTERRUPT_CONFIG interruptConfig;

                        MyTracePrint(VERBOSE, "\t\tInt type: MSI/MSI-X\n");
                        MyTracePrint(VERBOSE, "\t\tMessages: %d\n", resourceRaw->u.MessageInterrupt.Raw.MessageCount)

                        //
                        // Create WDFINTERRUPT object thereby connecting to the interrupt.
                        //
                        WDF_INTERRUPT_CONFIG_INIT(&interruptConfig,
                                                  MyIsr,
                                                  MyDpcForIsr);
 
                        interruptConfig.EvtInterruptEnable  = MyInterruptEnable;
                        interruptConfig.EvtInterruptDisable = MyInterruptDisable;
                        interruptConfig.InterruptTranslated = resourceTrans;
                        interruptConfig.InterruptRaw        = resourceRaw;

                        status = WdfInterruptCreate(devContext->WdfDevice,
                                            &interruptConfig,
                                            WDF_NO_OBJECT_ATTRIBUTES,
                                            &devContext->InterruptObjects[interruptResourcesFound]);

                        if(!NT_SUCCESS(status)) {

                            MyTracePrint(ERROR, "ConnectInterrupt failed? Status = 0x%lx\n", status);

                            goto done;
                        }

                    } else {

                        MyTracePrint(ERROR, "UNEXPECTED Int type LBI found\n");

                        //
                        // Win32: ERROR_INVALID_PARAMETER
                        //
                        status = STATUS_DEVICE_CONFIGURATION_ERROR;

                        goto done;
                    }

                    interruptResourcesFound++;

                    totalInterruptsFound += resourceRaw->u.MessageInterrupt.Raw.MessageCount;

                } else {
    
                    MyTracePrint(INFO,"********UNEXPECTED INTERRUPT RESOURCE FOUND (not used)... %d interrupts already found.\n", totalInterruptsFound);

                }

                break;
            }

ETA: Clarify that the code above is intended ONLY for MSI-X, and will not work for MSI (where you need to create on WDFINTERRUPT for each message you receive, unlike in MSI-X where you create one WDFINTERRUPT for each Interrupt Resource you receive)

6

Thanks Peter. Really appreciate the explanation.