Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTDEV

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Before Posting...

Please check out the Community Guidelines in the Announcements and Administration Category.

Setting Up MSIX Interrupts For a PCI Device [WDF]

ajitabhsajitabhs Member Posts: 19

Hi Guys,
This is a general question regarding WDF drivers written for a PCIe device which has multiple interrupt sources (for example multiple queues). In my interrupt service routine I would like to identify every source without reading the "interrupt status" register. The message code that we get in the ISR should give me the source of the interrupt, so that I do not need to go and read a status register from the hardware. Few questions:

  1. Can someone point me to the sample source code which does this?
  2. Any pointers to WDF documentation about this.
  3. What happens if there are two interrupts (hence two messages) at the same time? I think there will two interrupts with two messages. but please correct me if I am missing something here.

Any help highly appreciated.

Thanks
Ajitabh

Comments

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 14,163

    Why don't you want to read the "interrupt status" register? That's what it's for, for gosh sakes.

    In general, all of the interrupts for a single device are routed to the same CPU, so multiple MSI interrupts should be serialized. I doubt there is any guarantee of that.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,784
    edited November 22

    Why don't you want to read the "interrupt status" register

    Because it's a serializing operation. One of the goals of MSI/MSI-X is to make it so that you don't need to touch ANY device registers in your ISR (or, in many cases, any time after device setup/initialization). This avoids the slow-downs due to serialization. In high -speed devices/transfers, like the OP, it's usually one of my goals to not touch any device registers in the ISR. You use the MSI message number (for example) to tell you which set of memory resident data structures you need to process in your ISR (and/or DPC).

    all of the interrupts for a single device are routed to the same CPU

    Hmmmm... really? I've never heard this in all the years I've been writing drivers... and I'm not sure it matches my experience. Windows definitely connects devices to all CPUs... and I was under the impression that the ICH/PCH had "magic" to determine which CPU to which to route the interrupt from a given device (it's in one of the "colored" Intel books, IIRC).

    It's definitely not the case for MSI-X where the device can actually CHOOSE the CPU to which to target the interrupt.

    The message code that we get in the ISR should give me the source of the interrupt, so that I do not need to go and read a status register from the hardware

    Well, yes or no, depending. The prototype for your ISR is:

    BOOLEAN 
    DeviceInterrupt_EvtInterruptIsr(
        _In_ WDFINTERRUPT Interrupt,
        _In_ ULONG MessageID
        )
    

    Right? So, for MSI type interrupts, you just look at the MessageID in your ISR to determine which interrupt is being services. You'll get one Interrupt resource (passed to you in EvtDevicePrepareHardware) and you need to create one WDFINTERRUPT for each message thats assigned to you for that single Interrupt resource (see the u.MessageInterrupt.Raw.MessageCount field of the raw resource descriptor). The ISRs that you connect to this (one) interrupt resource will be called for all your MSIs. You differentiate using the MessageID field.

    Except... you specifically said MSI-X in your title, not MSI.

    For MSI-X things are slightly different. You'll get one interrupt resource (passed to you in EvtDevicePrepareHardware) for each MSI-X interrupt that you're granted. Therefore the ISR that's called will imply which interrupt was generated. You will need to connect these interrupts (calling WdfInterruptCreate) from your EvtDevicePrepareHardware Event Processing Callback.

    The sort of basic documentation for this is here... it could definitely be more helpful, as it leaves a lot of pragmatic things unaddressed. But my comments above should fill those missing things in sufficiently for you.

    Peter

    ETA: Clarify that you need to call WdfInterruptCreate for each MSI, which was not clear in the original post

    Post edited by Peter_Viscarola_(OSR) on

    Peter Viscarola
    OSR
    @OSRDrivers

  • ajitabhsajitabhs Member Posts: 19

    Thank you peter. This is really helpful. I will go through the documentation on this and will also look at samples. If you can recommend any samples that will be great.

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,784
    edited November 22

    I'm not sure why you're looking for a sample. There's really nothing to see, particularly in the ISR.

    The only even slightly tricky item is calling WdfInterruptCreate from your EvtDevicePrepareHardware, which is badly described in the docs, but which might want to look something like this:

                //
                // We expect MSI-X resources only.  The device does not fall-back to an LBI. 
                // NOTE: THE CODE BELOW DOES NOT WORK FOR MSI
                //
                case CmResourceTypeInterrupt: {
    
                    MyTracePrint(INFO,"Resource %lu: Interrupt\n", i);
    
                    if(totalInterruptsFound < MY_DEVICE_EXPECTED_NUMBER_OF_INTERRUPTS) {
    
                        resourceRaw = WdfCmResourceListGetDescriptor(Resources, i);
    
                        if(resourceTrans->Flags & CM_RESOURCE_INTERRUPT_MESSAGE) {
    
                            WDF_INTERRUPT_CONFIG interruptConfig;
    
                            MyTracePrint(VERBOSE, "\t\tInt type: MSI/MSI-X\n");
                            MyTracePrint(VERBOSE, "\t\tMessages: %d\n", resourceRaw->u.MessageInterrupt.Raw.MessageCount)
    
                            //
                            // Create WDFINTERRUPT object thereby connecting to the interrupt.
                            //
                            WDF_INTERRUPT_CONFIG_INIT(&interruptConfig,
                                                      PDAQ4Isr,
                                                      PDAQ4DpcForIsr);
    
                            interruptConfig.EvtInterruptEnable  = MyInterruptEnable;
                            interruptConfig.EvtInterruptDisable = MyInterruptDisable;
                            interruptConfig.InterruptTranslated = resourceTrans;
                            interruptConfig.InterruptRaw        = resourceRaw;
    
                            status = WdfInterruptCreate(devContext->WdfDevice,
                                                &interruptConfig,
                                                WDF_NO_OBJECT_ATTRIBUTES,
                                                &devContext->InterruptObjects[interruptResourcesFound]);
    
                            if(!NT_SUCCESS(status)) {
    
                                MyTracePrint(ERROR, "ConnectInterrupt failed? Status = 0x%lx\n", status);
    
                                goto done;
                            }
    
                        } else {
    
                            MyTracePrint(ERROR, "UNEXPECTED Int type LBI found\n");
    
                            //
                            // Win32: ERROR_INVALID_PARAMETER
                            //
                            status = STATUS_DEVICE_CONFIGURATION_ERROR;
    
                            goto done;
                        }
    
                        interruptResourcesFound++;
    
                        totalInterruptsFound += resourceRaw->u.MessageInterrupt.Raw.MessageCount;
    
                    } else {
    
                        MyTracePrint(INFO,"********UNEXPECTED INTERRUPT RESOURCE FOUND (not used)... %d interrupts already found.\n", totalInterruptsFound);
    
                    }
    
                    break;
                }
    
    

    ETA: Clarify that the code above is intended ONLY for MSI-X, and will not work for MSI (where you need to create on WDFINTERRUPT for each message you receive, unlike in MSI-X where you create one WDFINTERRUPT for each Interrupt Resource you receive)

    Post edited by Peter_Viscarola_(OSR) on

    Peter Viscarola
    OSR
    @OSRDrivers

  • ajitabhsajitabhs Member Posts: 19

    Thanks Peter. Really appreciate the explanation.

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. Sign in or register to get started.

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Writing WDF Drivers 24 January 2022 Live, Online
Internals & Software Drivers 7 February 2022 Live, Online
Kernel Debugging 21 March 2022 Live, Online
Developing Minifilters 23 May 2022 Live, Online