Interrupts never trigger ISR

I’ve been developing a KMDF driver for a new custom multiport serial device that uses the Xilinx XDMA for DMA. Mostly it’s fine, but for some reason I am completely unable to get an interrupt. I follow all the steps in Microsoft’s guides for creating an ISR DPC, I have compared it to a previous product we developed that works just fine. The interrupt resource shows up appropriately in PrepareHardware, I get no errors in the process at any step that indicates that the interrupt shouldn’t fire. The interrupts trigger just fine in Linux. I had the firmware developer make a temporary register that should cause an interrupt storm when I set a specific bit. I even downloaded some trial software that allowed me to see that interrupts were happening, but the ISR itself never triggers. I have DbgPrints in InterruptEnable and InterruptDisable telling me that the interrupt should be enabled. Nothing I do will cause the ISR to fire.

Here’s the code in DeviceAdd:

WDF_INTERRUPT_CONFIG_INIT(&interruptConfig, newproject_isr, NULL);
interruptConfig.ShareVector = WdfTrue;
interruptConfig.EvtInterruptEnable = EvtWdfInterruptEnable;
interruptConfig.EvtInterruptDisable = EvtWdfInterruptDisable;
WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&interruptAttributes, NEWPROJECT_PORT);

status = WdfInterruptCreate(port->device, &interruptConfig, &interruptAttributes, &port->interrupt);
if (!NT_SUCCESS(status)) {
	TraceEvents(TRACE_LEVEL_ERROR, TRACE_DEVICE, "WdfInterruptCreate failed %x\n", status);
	return 0;
}

At no point in any step of the process do I get an error.

Is there some magic trick I’m overlooking? Some pitfall I may not know about? Everything suggests the isr should fire, and the first thing I do in the isr is a debug print, and I never see a single one.

I’m tearing my hair out here, thanks.

ShareVector = WdfTrue - if this is intentional, then the ISR can be chained at the end of a queue. Is there a device that improperly claims the interrupt ahead of your device? Windows-driver-samples\setup\devcon implements a resources switch that dumps the IRQ:

Windows-driver-samples\setup\devcon\x64\Debug\devcon.exe help resources
Devcon Resources Command
Lists hardware resources of devices with the specified hardware or instance ID.
Valid on local and remote computers.
devcon.exe [-m:\\<machine>] resources <id> [<id>...]
Examples of <id>:
 *              - All devices
 PCI\*          - PCI devices with wildcards  (* matches anything)

That does not appear to be the issue. There are no other devices in the system, and even when I force it to WdfFalse (which only allows one port to be generated) it still behaves the same way. Also, devcon shows only the filter driver and the port drivers sharing the irq (19). Nothing else is using irq 19.

Thanks for the suggestion. Any other ideas?

Standard PCI interrupts (non-MSI) are always shared. The only question I have is about ‘port->device’. Is this a ‘multiport’ device, with a parent device and multiple child devices?

!arbiter 4 + !idt are a starting point.

Here's the output of both. I am unfamiliar with both arbiter and idt, and arbiter seems to have some strange indentation patterns (which I only mention because I'm unsure if it's relevant, if it is I can provide more context). fsccxmc-windows is my driver. It is a multiport device with 6 ports, 3 that will be part of fsccxmc-windows and 3 that will be part of a different driver. Neither of the drivers isrs trigger under any circumstance, and the other driver is currently uninstalled to reduce noise.

!arbiter 4
...
DEVNODE ffffbc05c4d58010 (PCI\VEN_10EE&DEV_7021&SUBSYS_000710EE&REV_00\4&1286464&0&0008)
Interrupt Arbiter "Mf Arbiter" at ffffa98c3c7e97e8
Allocated ranges:
0000000000000000 - 0000000000000012       00000000 
0000000000000013 - 0000000000000013 S
0000000000000013 - 0000000000000013 S     ffffbc05c7a35d40  (fsccxmc-windows)
0000000000000013 - 0000000000000013 S     ffffbc05c7a24d40  (fsccxmc-windows)
0000000000000013 - 0000000000000013 S     ffffbc05c7a13d40  (fsccxmc-windows)
0000000000000014 - ffffffffffffffff       00000000 
Possible allocation:
< none >

!idt:
...
70:	fffff80323415640 serial!SerialCIsrSw (KINTERRUPT ffff87011fb79640)
71:	fffff80323415648 fsccxmc_windows+0x5880 (KMDF) (KINTERRUPT ffff87011fb79280)
fsccxmc_windows+0x5880 (KMDF) (KINTERRUPT ffff87011fb79140)
fsccxmc_windows+0x5880 (KMDF) (KINTERRUPT ffff87011fb79000)
81:	fffff803234156c8 dxgkrnl!DpiFdoMessageInterruptRoutine (KINTERRUPT ffff87012e87fdc0)
...

Here's the output of both. I am unfamiliar with both arbiter and idt, and arbiter seems to have some strange indentation patterns (which I only mention because I'm unsure if it's relevant, if it is I can provide more context). fsccxmc-windows is my driver. It is a multiport device with 6 ports, 3 that will be part of fsccxmc-windows and 3 that will be part of a different driver. Neither of the drivers isrs trigger under any circumstance, and the other driver is currently uninstalled to reduce noise.

!arbiter 4
...
DEVNODE ffffbc05c4d58010 (PCI\VEN_10EE&DEV_7021&SUBSYS_000710EE&REV_00\4&1286464&0&0008)
Interrupt Arbiter "Mf Arbiter" at ffffa98c3c7e97e8
Allocated ranges:
0000000000000000 - 0000000000000012       00000000 
0000000000000013 - 0000000000000013 S
0000000000000013 - 0000000000000013 S     ffffbc05c7a35d40  (fsccxmc-windows)
0000000000000013 - 0000000000000013 S     ffffbc05c7a24d40  (fsccxmc-windows)
0000000000000013 - 0000000000000013 S     ffffbc05c7a13d40  (fsccxmc-windows)
0000000000000014 - ffffffffffffffff       00000000 
Possible allocation:
< none >

!idt:
...
70:	fffff80323415640 serial!SerialCIsrSw (KINTERRUPT ffff87011fb79640)
71:	fffff80323415648 fsccxmc_windows+0x5880 (KMDF) (KINTERRUPT ffff87011fb79280)
fsccxmc_windows+0x5880 (KMDF) (KINTERRUPT ffff87011fb79140)
fsccxmc_windows+0x5880 (KMDF) (KINTERRUPT ffff87011fb79000)
81:	fffff803234156c8 dxgkrnl!DpiFdoMessageInterruptRoutine (KINTERRUPT ffff87012e87fdc0)
...

Neither of those seem to indicate a problem, but I'm not familiar with either. This is a multiport device that has filter inf and 2 drivers, one being fsccxmc-windows (3 ports) and the other is current uninstalled to reduce spam. Neither the ISRs ever seem to trigger, regardless of which I have installed/uninstalled at any given time. I did notice !arbiter 4 has some odd indention. I only mention it because I'm unsure of its relevance. If it's relevant I can provide more context.

Output of !arbiter 4:
...
DEVNODE ffffd4025256e050 (PCI\VEN_10EE&DEV_7021&SUBSYS_000710EE&REV_00\4&1286464&0&0008)
Interrupt Arbiter "Mf Arbiter" at ffffc30d3dfe9368
Allocated ranges:
0000000000000000 - 0000000000000012 00000000
0000000000000013 - 0000000000000013 S
0000000000000013 - 0000000000000013 S ffffd40255f24d40 (fsccxmc-windows)
0000000000000013 - 0000000000000013 S ffffd40255f02d40 (fsccxmc-windows)
0000000000000013 - 0000000000000013 S ffffd40255eefd40 (fsccxmc-windows)
0000000000000014 - ffffffffffffffff 00000000
Possible allocation:
< none >

Output of !idt:
...
70: fffff80540015640 serial!SerialCIsrSw (KINTERRUPT ffff9c010a58b640)
71: fffff80540015648 fsccxmc_windows!fsccxmc_isr (KMDF) (KINTERRUPT ffff9c010a58b280)
fsccxmc_windows!fsccxmc_isr (KMDF) (KINTERRUPT ffff9c010a58b140)
fsccxmc_windows!fsccxmc_isr (KMDF) (KINTERRUPT ffff9c010a58b000)
81: fffff805400156c8 dxgkrnl!DpiFdoMessageInterruptRoutine (KINTERRUPT ffff9c010c1bfdc0)
...

Some low hanging fruits:

  • compare the ports’ PCI Configuration Space between Windows and Linux.
  • !pci 145 <Bus> <Device> <Function> for endpoint, bridge if present.
1 Like

I spent a little time trying to process the configuration space, and had a surprisingly difficult time finding information about what the Status and Command registers represent, but at least those two seem to indicate that there shouldn’t be any issues. I’m thinking the next step might be to implement a function in my driver that uses SetBusData in the BUS_INTERFACE_STANDARD structure to try and forcefully trigger an interrupt. Does that seem appropriate? The firmware dev implemented a register that should do that very thing, but it never seems to trigger the ISR in Windows.

I should mention that we’re currently using the Xilinx vendor/devid, but that will be changed before we ship anything. I figured that was implied, but maybe it was worth saying out loud.

0: kd> !pci 145 1 0 0

PCI Configuration Space (Segment:0000 Bus:01 Device:00 Function:00)
Common Header:
    00: VendorID       10ee Xilinx Corporation
    02: DeviceID       7021
    04: Command        0006 MemSpaceEn BusInitiate 
    06: Status         0010 CapList 
    08: RevisionID     00
    09: ProgIF         01 16450 Compatible
    0a: SubClass       00 Serial Controller
    0b: BaseClass      07 Simple Communications Controller
    0c: CacheLineSize  0010 BurstDisabled 
    0d: LatencyTimer   00
    0e: HeaderType     00
    0f: BIST           00
    10: BAR0           f7a00000
    14: BAR1           f78f0000
    18: BAR2           f7900000
    1c: BAR3           00000000
    20: BAR4           00000000
    24: BAR5           00000000
    28: CBCISPtr       00000000
    2c: SubSysVenID    10ee
    2e: SubSysID       0007
    30: ROMBAR         00000000
    34: CapPtr         40
    3c: IntLine        13
    3d: IntPin         04
    3e: MinGnt         00
    3f: MaxLat         00
Device Private:
    40: 00037001 00000008 00807005 00000000
    50: 00000000 00000000 00000000 00000000
    60: 00007011 00000000 00000000 00000000
    70: 00020010 012c8023 00092900 0043f042
    80: 10420040 00000000 00000000 00000000
    90: 00000000 00000016 00000000 00000006
    a0: 00010002 00000000 00000000 00000000
    b0: 00000000 00000000 00000000 00000000
    c0: 00000000 00000000 00000000 00000000
    d0: 00000000 00000000 00000000 00000000
    e0: 00000000 00000000 00000000 00000000
    f0: 00000000 00000000 00000000 00000000
Capabilities:
    40: CapID          01 PwrMgmt Capability
    41: NextPtr        70
    42: PwrMgmtCap     0003 Version=3
    44: PwrMgmtCtrl    0008 DataScale:0 DataSel:0 D0 

    70: CapID          10 PCI Express Capability
    71: NextPtr        00
    72: Express Caps   0002 (ver. 2) Type:Endpoint
    74: Device Caps    012c8023
    78: Device Control 2900 bcre/flr MRR:512 NS ap pf ET MP:128 ro ur fe nf ce
    7a: Device Status  0009 tp ap UR fe nf CE 
    7c: Link Caps      0043f042
    80: Link Control   0040 es CC rl ld RCB:64 ASPM:None 
    82: Link Status    1042 SCC lt lte NLW:x4 LS:2.5 
    94: DeviceCaps2    00000016 CTR:6 CTDIS arifwd aor aoc32 aoc64 cas128 noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0
    98: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom ltr OBFF:0 eetlp
    84: Slot Caps      00000000 
    88: Slot Control   0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab 
    8a: Slot Status    0000 dlsc eis pds hpi cc pdc ms pfd ab 
    8c: Root Control   0000 pmei fs nfs cs 
    8e: Root Caps      0000 
    90: Root Status    00000000 pmep pmes ID:0 

Enhanced Capabilities:
    100: CapID         0001 Advanced Error Reporting Capability
         Version       1
         NextPtr       1f0
    104: UncorrectableErrorStatus   00000000  dlpe sde ptlp fcpe ct ca mtlp ecrc ur acsv uie mcbtlp aeb tpb
    108: UncorrectableErrorMask     00400000  dlpe sde ptlp fcpe ct ca mtlp ecrc ur acsv UIE mcbtlp aeb tpb
    10c: UncorrectableErrorSeverity 00462030  DLPE SDE ptlp FCPE ct ca MTLP ecrc ur acsv UIE mcbtlp aeb tpb
    110: CorrectableErrorStatus     00002000  re btlp bdllp rnr rtt ANFE cie hlo
    114: CorrectableErrorMask       0000e000  re btlp bdllp rnr rtt ANFE CIE HLO
    118: CapabilitiesAndControl     00000000  FirstErr:0 ecrcgcap ecrcgen ecrcccap ecrccen mhrcap mhren tplp
    11c: HeaderLog:    00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  
         Fmt           0 (3 DW Header, no data)
         Type          0 (MRd)
         Length        0 (0 bytes)
         RID           0:0:0
         Tag           0
         Attributes    0 (Default Ordering)
         Traffic Class 0
         Address       00000000

    1f0: CapID         0002 Virtual Channel Capability
         Version       1
         NextPtr       000
    01f4: Port VC Capability 1        00000000
    01f8: Port VC Capability 2        31000000
    01fc: Port VC Control             0000
    01fe: Port VC Status              0000
    0200: VC Resource[0] Cap          00000000
    0204: VC Resource[0] Control      80000001
    020a: VC Resource[0] Status       0000

My gut says you should be setting your INF to use MSI interrupts and are not doing so. Are you using MSI on Linux? There’s no reason to be using legacy interrupts with XDMA and it would not surprise me if it’s buggy on the Xilinx side. See this link for the INF setup:

Also see the setup code that @Peter_Viscarola_OSR posted for me a while back in this post:

EDIT:

Just reread your post and notice you are doing this in DeviceAdd? That is not the correct place. You need to do it in the EvtDevicePrepareHardware callback when the OS gives you information about the interrupt. You can’t simply add an interrupt and not tie it to an actual interrupt. I will see if I can dig up some sharable code tomorrow.

Are you sure? The WdfInterruptCreate docs (and the working drivers for our other devices) suggest it’s perfectly acceptable to call WdfInterruptCreate in DeviceAdd:

I do wonder if there’s something I’m supposed to be doing in PrepareHardware to hook the legacy interrupt into the ISR, but no such thing is done in our older drivers and I can’t find any reference suggesting it’s necessary, but if there is something I’m supposed to do I’d be happy to hear it.

@Josh1 , here’s my code anonymized and trimmed of the all the stuff specific to my card. This works for both a legacy interrupt or a single MSI vector. I prefer to separate my interrupt initialization into a separate function.

_Use_decl_annotations_
NTSTATUS
MyPrepareHardware(
    WDFDEVICE Device,
    WDFCMRESLIST ResourcesRaw,
    WDFCMRESLIST ResourcesTranslated
)
{
    PCM_PARTIAL_RESOURCE_DESCRIPTOR raw, descriptor;
    NTSTATUS status = STATUS_SUCCESS;
    ULONG i{};

    PAGED_CODE();

    TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_DEVICE, "%!FUNC! Entry");

    auto deviceContext = DeviceGetContext(Device);

    for (i = 0; i < WdfCmResourceListGetCount(ResourcesTranslated); i++) {
        raw = WdfCmResourceListGetDescriptor(ResourcesRaw, i);
        descriptor = WdfCmResourceListGetDescriptor(ResourcesTranslated, i);

        switch (descriptor->Type) {
        case CmResourceTypeInterrupt:
            status = MyInterruptInitialize(Device, raw, descriptor);
            if (!NT_SUCCESS(status)) {
                TraceEvents(TRACE_LEVEL_ERROR, TRACE_DEVICE,
                    "InterruptInitialize failed %!STATUS!", status);
                return status;
            }
            break;
        case CmResourceTypeMemory:
            // handle BARs here
            break;
        default:
            break;
        }
    }

    // additional initialization

    TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_DEVICE, "%!FUNC! Exit");

    return status;
}

_Use_decl_annotations_
NTSTATUS
MyInterruptInitialize(
    WDFDEVICE Device,
    PCM_PARTIAL_RESOURCE_DESCRIPTOR InterruptRaw,
    PCM_PARTIAL_RESOURCE_DESCRIPTOR InterruptTranslated
)
{
    WDF_OBJECT_ATTRIBUTES interruptAttributes;
    WDF_INTERRUPT_CONFIG interruptConfig;
    NTSTATUS status = STATUS_SUCCESS;

    PAGED_CODE();

    TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_DEVICE, "%!FUNC! Entry");

    auto deviceContext = DeviceGetContext(Device);

    if (CM_RESOURCE_INTERRUPT_MESSAGE & InterruptTranslated->Flags) {
        TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_DEVICE,
            "Message Interrupt Affinity 0x%0llx, Vector 0x%0x, Level 0x%0x, MessageCount %hu",
            InterruptTranslated->u.MessageInterrupt.Translated.Affinity,
            InterruptTranslated->u.MessageInterrupt.Translated.Vector,
            InterruptTranslated->u.MessageInterrupt.Translated.Level,
            InterruptRaw->u.MessageInterrupt.Raw.MessageCount);
    }
    else {
        TraceEvents(
            TRACE_LEVEL_INFORMATION, TRACE_DEVICE, "Legacy Interrupt level: 0x%0x, Vector: 0x%0x",
            InterruptTranslated->u.Interrupt.Level, InterruptTranslated->u.Interrupt.Vector);
    }

    WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&interruptAttributes, MY_ISR_CONTEXT);
    WDF_INTERRUPT_CONFIG_INIT(&interruptConfig, MyInterruptIsr, MyInterruptDpc);

    interruptConfig.InterruptRaw = InterruptRaw;
    interruptConfig.InterruptTranslated = InterruptTranslated;

    status = WdfInterruptCreate(Device, &interruptConfig, &interruptAttributes,
        &deviceContext->InterruptObject[0]);

    if (NT_SUCCESS(status)) {
        // Get the context
        auto isrContext = MyIsrGetContext(deviceContext->InterruptObject[0]);

        // Save the interrupt type
        isrContext->MessageSignaled = ((CM_RESOURCE_INTERRUPT_MESSAGE & InterruptTranslated->Flags) != 0);
    }
    else {
        TraceEvents(TRACE_LEVEL_ERROR, TRACE_DEVICE, "WdfInterruptCreate failed %!STATUS!", status);
    }

    TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_DEVICE, "%!FUNC! Exit");

    return status;
}

Thanks for the help!

It took me a while to figure it out, and I still haven’t resolved every issue, but I can now make interrupts happen.

The issue is that in order to use the Xilinx DMA interrupts, you need to write the interrupt pin being used into the userVector and channelVector registers of the irq block. This was something I noticed somewhat early, but didn’t seem to have any effect, so I didn’t think much about it

The second issue that I discovered as I was crawling assorted code hosting websites looking for anything that might offer insight into why the interrupts weren’t working was in the form of a comment in someone else’s drivers before writing into the *vector registers:

// Windows: A=1, B=2, C=3, D=4
// XDMA: A=0, B=1, C=2, D=3

Sure enough, if I subtracted 1 from the interrupt pin found in the configuration space, and wrote that into the appropriate Xilinx DMA registers instead, I am suddenly able to get interrupts.

I now have interrupt storm issues, but that’s a different issue so I’m going to close this question out. Hopefully someone will find it useful.

Thanks again, everyone.

Are you sure you have an interrupt storm or are you just not handling the interrupt properly? Usually legacy interrupts are level sensitive. So if your ISR isn’t clearing all interrupts before returning it will just trigger again.

I have not used XDMA but highly suggest looking at using MSI interrupts. We had an old design which still used legacy interrupts and a year ago the new servers a customer started buying stopped supporting them and we had to do an update to support MSI. I believe the PCIe spec says you must support MSI and legacy interrupts are optional.

I was probably using improper terminology. I was/am not handling the interrupts properly.

As much as I may want to try something new, I am stuck with legacy interrupts for the moment. This is a custom project and time is critical, so there isn’t much space for trying new things.

Just to note something for the archives:

You can create your WDFINTERRUPT object in DeviceAdd if you always know you’re only going to get one interrupt. It’s no different than doing in PrepareHardware in that case though.

That has not been my experience with PCIe connected FPGAs. Perhaps something I will have to try again though at some point.

I get that. However if you find a few free minutes just try adding the lines to the INF file and see if it “just works”. I would think the XDMA core would have MSI enabled by default.

It’s certainly supported and works:

A driver typically creates framework interrupt objects in its EvtDriverDeviceAdd callback function. A driver can also create interrupt objects from its EvtDevicePrepareHardware callback function.

Creating an Interrupt Object - Windows drivers | Microsoft Learn

Though like I said there’s no advantage over doing it in PrepareHardware when you’re only getting one interrupt.

Seeing as the topic at hand is interrupts, I’ve been reading about MSI because while my legacy interrupts are working now there seems to be an occasional missed interrupt (somehow) so I was hoping that implementing MSI may resolve that. But after reading about it, I am brought to something I’m not sure if I fully grasp:

I understand from the post linked by GrimBeaver that the drivers may be granted less than the expected MSI interrupts, including the possibility of zero (0) MSI resources, even with MSI enabled in the INF and enabled in the device, but is that a decision done by the OS? If so, does that not mean I absolutely need to have the ability to properly handle legacy interrupts? Or am I misunderstanding something?