Interrupt service routine not called by the framework

Hello all,

I spent three days studying network and forum topics but i was not able to solve my problem until know.

I am quite new in writing windows drivers, I am preparing KMDF driver for my device connected to the PCIe. I am able to read and write to the memory area of the device, read BARs data, information about interrupts etc.

I have prepared code in WDF to manage with interrupts (old line interrupt triggered by low level) raised by my device. I see in WinDbg that FPGAdrvEvtInterruptEnable and FPGAdrvEvtInterruptDisable functions are called by framework. I have problem that my InterruptServiceRoutine is not called by framework.

What I have checked:

  • with oscilloscope interrupt line - interrupt is generated correctly by the hardware
  • with WinDbg - my ISR looks correctly registered in the framework:

Dumping IDT: fffff8024ee61000

00: fffff8024b601c00 nt!KiDivideErrorFault
01: fffff8024b601f40 nt!KiDebugTrapOrFault Stack = 0xFFFFF8024EE9F000
02: fffff8024b602440 nt!KiNmiInterrupt Stack = 0xFFFFF8024EE91000
03: fffff8024b602900 nt!KiBreakpointTrap

82: fffff8024b5f9d90 FPGAdrv!FPGAdrvEvtInterruptIsr (KMDF) (KINTERRUPT ffffde80632a08c0)

90: fffff8024b5f9e00 msgpioclx!GpioClxEvtInterruptIsr (KMDF) (KINTERRUPT ffffde80632a0c80)
.
0: kd> dt nt!_KINTERRUPT ffffde80632a08c0
+0x000 Type : 0n22
+0x002 Size : 0n288
+0x008 InterruptListEntry : _LIST_ENTRY [ 0x0000000000000000 - 0x0000000000000000 ]
+0x018 ServiceRoutine : 0xfffff8024d643c30 unsigned char Wdf01000!FxInterrupt::_InterruptThunk+0 +0x020 MessageServiceRoutine : (null) +0x028 MessageIndex : 0 +0x030 ServiceContext : 0xffff868e9e5185f0 Void
+0x038 SpinLock : 0
+0x040 TickCount : 0
+0x048 ActualLock : 0xffff868e9e518690 -> 0 +0x050 DispatchAddress : 0xfffff8024b5f8c80 void nt!KiInterruptDispatch+0
+0x058 Vector : 0x82
+0x05c Irql : 0x8 ‘’
+0x05d SynchronizeIrql : 0x8 ‘’
+0x05e FloatingSave : 0 ‘’
+0x05f Connected : 0x1 ‘’
+0x060 Number : 0
+0x064 ShareVector : 0x1 ‘’
+0x065 EmulateActiveBoth : 0 ‘’
+0x066 ActiveCount : 0
+0x068 InternalState : 0n4
+0x06c Mode : 0 ( LevelSensitive )
+0x070 Polarity : 0 ( InterruptPolarityUnknown )
+0x074 ServiceCount : 0
+0x078 DispatchCount : 0
+0x080 PassiveEvent : (null)
+0x088 TrapFrame : (null)
+0x090 DisconnectData : (null)
+0x098 ServiceThread : (null)
+0x0a0 ConnectionData : 0xffff868ea5e0ea30 _INTERRUPT_CONNECTION_DATA +0x0a8 IntTrackEntry : 0xffff868ea78c9a50 Void
+0x0b0 IsrDpcStats : _ISRDPCSTATS
+0x110 RedirectObject : (null)
+0x118 PhysicalDeviceObject : 0xffff868e`9e5f9060 Void
.
Processor 0 (0, 0):
Device Object: 0000000000000000
Current IDT Allocation:
0000000000000000 - 0000000000000036 00000000 A:0000000000000000 IRQ(GSIV):10
0000000000000050 - 0000000000000050 D ffff868e9e62a360 (pci) A:ffffae02235c9af0 IRQ(GSIV):fffffffe
0000000000000051 - 0000000000000051 S B ffff868e9e5e1060 (sdbus) A:ffffae02235c98b0 IRQ(GSIV):27
0000000000000060 - 0000000000000060 D ffff868e9e61b120 A:ffffae02235c9910 IRQ(GSIV):2
0000000000000061 - 0000000000000061 D ffff868e9e63d360 (USBXHCI) A:ffffae02235ca030 IRQ(GSIV):fffffff8
0000000000000070 - 0000000000000070 D ffff868e9e5cbcf0 (Serial) A:ffffae02235ca060 IRQ(GSIV):3
0000000000000071 - 0000000000000071 D ffff868e9e628360 (storahci) A:ffffae02235ca120 IRQ(GSIV):fffffff9
0000000000000080 - 0000000000000080 D ffff868e9e5d4cf0 (Serial) A:ffffae02235c9dc0 IRQ(GSIV):4
0000000000000081 - 0000000000000081 D ffff868e9e639360 (pci) A:ffffae02235ca150 IRQ(GSIV):fffffffa
0000000000000082 - 0000000000000082 S B ffff868e9e5f9060 (FPGAdrv) A:ffffae02235ca180 IRQ(GSIV):16
0000000000000090 - 0000000000000090 S B ffff868e9e3edc60 (iaLPSS2i_GPIO2_BXT_P) A:ffffae02235ca330 IRQ(GSIV):e
0000000000000091 - 0000000000000091 D ffff868e9e5d1360 (pci) A:ffffae02235ca360 IRQ(GSIV):fffffffb
0000000000000092 - 0000000000000092 S B ffff868e9e5e0360 (HDAudBus) A:ffffae02235ca1e0 IRQ(GSIV):19
00000000000000a0 - 00000000000000a0 D ffff868e9e3eece0 A:ffffae02235ca0c0 IRQ(GSIV):8
00000000000000a1 - 00000000000000a1 D ffff868e9e5cd360 (pci) A:ffffae02235ca2a0 IRQ(GSIV):fffffffc
00000000000000a2 - 00000000000000a2 D ffff868e9e5de360 (igfx) A:ffffae02235ca300 IRQ(GSIV):fffffff6
00000000000000b0 - 00000000000000b0 S B ffff868e9df2cd00 (ACPI) A:ffffae02235c9e80 IRQ(GSIV):9
00000000000000b1 - 00000000000000b1 D ffff868e9e5e6360 (pci) A:ffffae02235c9eb0 IRQ(GSIV):fffffffd
00000000000000b2 - 00000000000000b2 D ffff868e9e626360 (MEIx64) A:ffffae02235cade0 IRQ(GSIV):fffffff7
00000000000000cd - 00000000000000ff 00000000 A:0000000000000000 IRQ(GSIV):0
0000000000000200 - ffffffffffffffff 00000000 A:0000000000000000 IRQ(GSIV):0
.
0: kd> dt acpi!_IDT_ASSIGNMENT_SET ffffae02235ca180
+0x000 Affinity : _GROUP_AFFINITY
+0x010 Gsiv : 0x16
+0x014 BaseIdtEntry : 0x82
+0x018 Length : 1
.
0: kd> !devstack ffff868e9e5f9060
!DevObj !DrvObj !DevExt ObjectName
ffff868ea3b65b60 \Driver\FPGAdrv ffff868ea4d63fb0
ffff868e9e5cfe00 \Driver\ACPI ffff868e9e583400
ffff868e9e5f9060 \Driver\pci ffff868e9e5f91b0 NTPNP_PCI0014
!DevNode ffff868e9e61fc90 :
DeviceInst is “PCI\VEN_1B05&DEV_0007&SUBSYS_00071B06&REV_00\0000000101000A3500”
ServiceName is “FPGAdrv”
.
Interrupt registration code is rather simple:

            WDF_INTERRUPT_CONFIG interruptConfig;
            //
            // Create WDFINTERRUPT object.
            //
            WDF_INTERRUPT_CONFIG_INIT(&interruptConfig,
                FPGAdrvEvtInterruptIsr,
                FPGAdrvEvtInterruptDpc);

            //
            // These first two callbacks will be called at DIRQL.  Their job is to
            // enable and disable interrupts.
            //
            interruptConfig.EvtInterruptEnable = FPGAdrvEvtInterruptEnable;
            interruptConfig.EvtInterruptDisable = FPGAdrvEvtInterruptDisable;
            interruptConfig.InterruptTranslated = descriptor;
            interruptConfig.InterruptRaw = WdfCmResourceListGetDescriptor(ResourcesRaw, i);

            status = WdfInterruptCreate(Device,
                &interruptConfig,
                WDF_NO_OBJECT_ATTRIBUTES,
                &deviceContext->WdfInterrupt);

            if (!NT_SUCCESS(status))
            {
                TraceEvents(TRACE_LEVEL_ERROR, TRACE_DEVICE,
                    "WdfInterruptCreate failed: %!STATUS!\n", status);
                DbgPrint("Interrupt not registered %x\n", status);
                return status;
            }

            DbgPrint("Interrupt registered sucessfully %x\n", status);

            TraceEvents(TRACE_LEVEL_VERBOSE, TRACE_DEVICE,
                "Interrupt level: 0x%0x, Vector: 0x%0x\n",
                descriptor->u.Interrupt.Level,
                descriptor->u.Interrupt.Vector);

I don’t have any idea what else can I check ? I have prepared this code based on the Microsoft pcidrv example, maybe i missed something ?

First, please verify: This is for a Line Based Interrupt, and your device has registered for JUST that single line based interrupt, correct? You’re not enabling MSI or MSI-X via registry entries, and there are no MSI or MSI-X Capabilities registered for your device? Is that all correct?

If that is correct, move your call to WdfInterruptCreate from EvtDevicePrepareHardware to EvtDriverDeviceAdd. Create one WDFINTERRUPT. You will not specify any resources here.

Assuming all the above is correct, WDF will connect your one Line Based Interrupt to the one WDF interrupt you’ve created.

Another thing to do is after EvtDevicePrepareHardware bus run, dump the log with !wdfkd.WDFLOGDUMP and read the output carefully. Look for any diagnostics related to interrupts (they’ll be in plain English). Make sure you’ve enabled WDF Verifier with Verbose logging, and are running with Windows Driver Verifier enabled.

Peter

1 Like

Thank you very much for your answer. Yes, I have only line interrupt in my device and I am not registering any MSI.

I moved interrupt object creation to the EvtDriverDeviceAdd but it didn’t solve my problem yet. I have enabled all additional logs (thanks for the suggestion).

Only one thing connected to the interrupt which I found is this line:

29: FxInterrupt::AssignResources - Is MSI? 0, MSI-ID 0, AffinityPolicy WdfIrqPolicyOneCloseProcessor, Priority WdfIrqPriorityUndefined, Group 0, Affinity 0x3, Irql 0x8, Vector 0x82

I was trying also to set up policy and priority but it didn’t change anything in calling my ISR.

btw. am I able to check somehow if CPU see my interrupt from hardware ?

I was trying also to set up policy and priority but it didn’t change anything in calling my ISR.

Don’t do ANY of that shit. Take out anything and everything related to your management of interrupts, except for your EvtDeviceInterruptEnable/Disable callbacks, your Dpc and ISR callbacks, EvtDevicePrepareHardware (where you now do nothing at all regarding your interrupts), and the creation of the WdfInterruptObject in EvtDriverDeviceAdd.

That logging output looks like things are all good…

am I able to check somehow if CPU see my interrupt

No. I mean… not without a PCI Bus Analyzer, which I don’t think was the intent of your question :wink:

SOMEbody needs to make an affordable PCI Bus Analyzer… I’ve wanted one myself multiple times in the last two weeks.

In your initial post, you wrote this:

with oscilloscope interrupt line - interrupt is generated correctly by the hardware

This is a BIT confusing… because you say you’re on an Express bus. And in that case, there IS no interrupt line. So, I’m a bit confused.

Peter

1 Like

Maybe he means the source interrupt signal before the PCIe message generation logic.

One doesn’t need to buy a whole cow for a glass of milk.
We’ve rented a LeCroy analyzer some time ago; it was very expensive but came with set-up assistance of their engineer.
The guy was extremely helpful, not only he saved us time to connect the analyzer, he also looked at the data and immediately spotted some bugs.
So it was well worth the price.

This is a BIT confusing… because you say you’re on an Express bus. And in that case, there IS no interrupt line. So, I’m a bit confused.

Thanks for the suggestion, I have checked what our hardware engineers done. They connected from FPGA to CPU PCIe bus but interrupt implemented on separate GPIO line. Now i need to manage with this type of setup.

I could implement second driver for GPIO controller and pass info from interrupt to my driver but it will be no more interrupt based signal. Additional delays etc.

Second option which I see is to try to conect to interrupt controller directly from my PCIe device driver and configure APIC to work with interrupt from interrupt capable GPIO. Here I am not sure hot to do proper connection to the APIC from my driver.

Neither of those options sound very reasonable to me. I don’t think I could make either one of them work in a way that’s architecturally sound, personally.

You have an FPGA…. Does it not include sufficient logic to send/receive data from the PCIe bus? If it does not, I’m not sure what to tell you…. If it does, you have everything you need to generate an MSI.

Peter

I fully agree, the problem is that because of different reasons FPGA firmware is untouchable :confused:

because of different reasons FPGA firmware is untouchable

And the board design is fixed? If so, you have what is effectively an unsupportable device design. Or, to be more clear, a device design that will not support interrupts. Generating GPIO interrupts from an Express device is… just plain silly. I’m not even sure if, technically, you can make that work on Windows. I don’t expect the PCI bus driver would understand GPIO interrupts. And I don’t expect any of ACPI (including the Resource Hub) to understand GPIO resources associated with a device that has PCIe resources.

Please tell me this is some sort of embedded system… where you can fool-around with stuff and not be concerned about other things in the configuration changing.

Peter

FPGA firmware is untouchable.

Come now. That is NEVER literally true. Every hardware engineer knows their designs are not flawless.

What may be true is that you, as the software engineer, are afraid to raise an objection at this point. However, that is your job. Gather your evidence and make a PowerPoint. Your FPGA team has produced a fatally flawed design that cannot be made to work reliably in Windows. It needs to be fixed. Better now then after it has been released to manufacturing. And if they released to manufacturing without having a driver, then maybe they deserve what they get. Without software support, and FPGA design is just a pile of power-sucking metal. (And my partner, the EE, hates it when I raise that point.)

Peter wrote:

Please tell me this is some sort of embedded system… where you can fool-around with stuff and not be concerned about other things in the configuration changing.

I did enough screwy drivers for embedded, but I would run away from this one. Bottom line is the FPGA designer produced a piece of garbage.

I’m going to agree with everyone on here …

a) PCIe is PCIe; it’s not very hard to get good IP in FPGA libraries these days, some of it even for free (I use the Numato Tagus for PCIe FPGA development [ https://numato.com/product/tagus-artix-7-pci-express-development-board/ ] and for that entry level FPGA Vivado has PCIe 2.0 IP for free. There’s absolutely no excuse for not building the FPGA the right way … not some bizarre “bus with a wire” design

b) It’s good that you have a GPIO exposed on the FPGA, but not for what you might think … I always design in a GPIO that can be toggled with a BAR register for “blinky LED” sorts of things from the driver (a “blinky LED” is the hardware equivalent of “Hello World” for software) … but it’s only for that, never for controlling something and definitely never for IO’s (and no, the PCIe bus won’t have a clue what to do with a level interrupt, that will go into a different bus altogether)

c) @“Peter_Viscarola_(OSR)” is also correct, KMDF will simplify all of the interrupt stuff in a very easy WDFInterrupt object (a few more lines for MSI interrupts), you don’t need any of that WDM stuff in there. There are good samples on GitHub of this as well as discussions in prior years on this site about hooking this up

d) Without question your hardware design is irretrievably broken. If it were me I would communicate gently but firmly to your manager that the EE should be strongly encouraged to pursue other career options outside of anything relating to hardware, see if you can connect a JTAG programmer to it somewhere, bulk erase the whole thing and start from scratch; again, there are good PCIe libraries out there. If they insist you carry on then tell them it will take not weeks, not months but years of work and you’re more than happy to disappear into a cave until 2026

e) @“Peter_Viscarola_(OSR)” Yep, a PCIe analyzer is really high up on my wish list but until that Powerball comes in (or a good analyzer is less than the cost of an house in most of the country) it’s just me and the “blinky LED” … :slight_smile:

Come now. That is NEVER literally true.

Your FPGA team has produced a fatally flawed design

It might not be “his” FPGA… They might have an FPGA they sourced from somewhere out of house, that they’re building onto a board.

OK… so, I agree, that’s rarely true. But, it happens. I had a client who had an FPGA that did some processing on particular device, PLUS a (reasonably expensive) PLX local bus to PCIe bridge chip. Government contract, of course. :wink:

Peter

Thank you for your comments, like Peter said it is not internal design. Redesign is other big topic which i dont want to introduce here to not lose main goal.

As a proof of concept for my managment I have modified FPGA design and rised PCIe legacy line interrupt there. My driver imediatelly discovered it.
In the ISR routine I have disabled interrupt, acknowledged it (PCIe FPGA memory) and returned TRUE to the framework.

Now, the thing is that ISR is called again and agin. I pretty sure that it is not from other device - until I will rise interrupt from PCIe FPGA this ISR is never called. Do I also need to inform framework that i managed with interrupt in other way than returning TRUE value ? Is it possible to get directly source of interrupt to verify it ? At this moment I am using example where interrupt is verified in ISR by reading status from device memory.

As ISR is called all the time my OS is almost hanging :confused:

Hm, I don’t know the details, but to me it sounds like even if you acknowledge the IRQ, you don’t clear the condition (in the FPGA) that causes the IRQ, so another IRQ is immediately generated after the previous one was handled. I.e. like a level-triggered IRQ, where the level is not cleared when the IRQ is handled.

Well… traditionally, for Line Based Interrupts (which are level triggered), your ISR needs to tell the device to stop asserting the interrupt. You also need a register bit that indicates that it was your device that requested the interrupt, so you know whether to return TRUE or FALSE.

Until your device stops interrupting, you’l keep getting interrupts. If you’re returning TRUE and your ISR keeps getting called, that interrupt has remained asserted.

Is it possible to get directly source of interrupt to verify it ?

Well, that’s what you do when you check the “interrupt request” bit set by your device when it asserts the interrupt request line. The OS doesn’t know who’s asserting that line or indeed how many devices are asserting it.

I’ll bet your device is still asserting the interrupt line.

Of course this is Express, so all this “assert” and “line” stuff is virtual and emulated.

Peter

I have set all this things in the device. Interrupt has been acknowledged and interrupt enable flag has been also desactivated.

It sounds that it is more connected to the device than Windows driver so I switch to FPGA side and debug it.

Thanks a lot for the help, that’s let me to diagnose problem and I have hope finally i will finish with good design.

Problem solved.

e) @“Peter_Viscarola_(OSR)” Yep, a PCIe analyzer is really high up on my wish list but until that Powerball comes in (or a good analyzer is less than the cost of an house in most of the country) it’s just me and the “blinky LED” … :slight_smile:

Actually used one is not so expensive depending on the requirements. Bigger cost will be additional proper cabling.

https://www.ebay.com/itm/233031634795?hash=item3641c48f6b:g:ld0AAOSwnG1cBW~E

LOL… yeah, good luck with a piece of complex, expensive, equipment from eBay. And… which version of PCIe will that support?

And, yes…. Still very expensive when you price out the cost of the necessary Interposer.

When I last checked, I couldn’t get a good PCIe V3 x4 (even) analyzer and “PCIe riser slot” type interposer for less than US$100K. And then… consider that if you had a device to debug (like and FPGA) that was soldered down on the main board and not on an add-in card, you’d need an additional Interposer for that (and the soldering skills to attach it without toasting the board). Beyond us here at OSR, for sure.

Peter

PCIe level TLP debugging is way beyond my abilities to diagnose, fix (or even comprehend …) … I depend on the FPGA IP’s in various libraries to get things right, if there are problems I’ll just switch to a different IP …