Interrupt service routine not called by the framework

Thank you very much for your answer. Yes, I have only line interrupt in my device and I am not registering any MSI.

I moved interrupt object creation to the EvtDriverDeviceAdd but it didn’t solve my problem yet. I have enabled all additional logs (thanks for the suggestion).

Only one thing connected to the interrupt which I found is this line:

29: FxInterrupt::AssignResources - Is MSI? 0, MSI-ID 0, AffinityPolicy WdfIrqPolicyOneCloseProcessor, Priority WdfIrqPriorityUndefined, Group 0, Affinity 0x3, Irql 0x8, Vector 0x82

I was trying also to set up policy and priority but it didn’t change anything in calling my ISR.

btw. am I able to check somehow if CPU see my interrupt from hardware ?

I was trying also to set up policy and priority but it didn’t change anything in calling my ISR.

Don’t do ANY of that shit. Take out anything and everything related to your management of interrupts, except for your EvtDeviceInterruptEnable/Disable callbacks, your Dpc and ISR callbacks, EvtDevicePrepareHardware (where you now do nothing at all regarding your interrupts), and the creation of the WdfInterruptObject in EvtDriverDeviceAdd.

That logging output looks like things are all good…

am I able to check somehow if CPU see my interrupt

No. I mean… not without a PCI Bus Analyzer, which I don’t think was the intent of your question :wink:

SOMEbody needs to make an affordable PCI Bus Analyzer… I’ve wanted one myself multiple times in the last two weeks.

In your initial post, you wrote this:

with oscilloscope interrupt line - interrupt is generated correctly by the hardware

This is a BIT confusing… because you say you’re on an Express bus. And in that case, there IS no interrupt line. So, I’m a bit confused.

Peter

1 Like

Maybe he means the source interrupt signal before the PCIe message generation logic.

One doesn’t need to buy a whole cow for a glass of milk.
We’ve rented a LeCroy analyzer some time ago; it was very expensive but came with set-up assistance of their engineer.
The guy was extremely helpful, not only he saved us time to connect the analyzer, he also looked at the data and immediately spotted some bugs.
So it was well worth the price.

This is a BIT confusing… because you say you’re on an Express bus. And in that case, there IS no interrupt line. So, I’m a bit confused.

Thanks for the suggestion, I have checked what our hardware engineers done. They connected from FPGA to CPU PCIe bus but interrupt implemented on separate GPIO line. Now i need to manage with this type of setup.

I could implement second driver for GPIO controller and pass info from interrupt to my driver but it will be no more interrupt based signal. Additional delays etc.

Second option which I see is to try to conect to interrupt controller directly from my PCIe device driver and configure APIC to work with interrupt from interrupt capable GPIO. Here I am not sure hot to do proper connection to the APIC from my driver.

Neither of those options sound very reasonable to me. I don’t think I could make either one of them work in a way that’s architecturally sound, personally.

You have an FPGA…. Does it not include sufficient logic to send/receive data from the PCIe bus? If it does not, I’m not sure what to tell you…. If it does, you have everything you need to generate an MSI.

Peter

I fully agree, the problem is that because of different reasons FPGA firmware is untouchable :confused:

because of different reasons FPGA firmware is untouchable

And the board design is fixed? If so, you have what is effectively an unsupportable device design. Or, to be more clear, a device design that will not support interrupts. Generating GPIO interrupts from an Express device is… just plain silly. I’m not even sure if, technically, you can make that work on Windows. I don’t expect the PCI bus driver would understand GPIO interrupts. And I don’t expect any of ACPI (including the Resource Hub) to understand GPIO resources associated with a device that has PCIe resources.

Please tell me this is some sort of embedded system… where you can fool-around with stuff and not be concerned about other things in the configuration changing.

Peter

FPGA firmware is untouchable.

Come now. That is NEVER literally true. Every hardware engineer knows their designs are not flawless.

What may be true is that you, as the software engineer, are afraid to raise an objection at this point. However, that is your job. Gather your evidence and make a PowerPoint. Your FPGA team has produced a fatally flawed design that cannot be made to work reliably in Windows. It needs to be fixed. Better now then after it has been released to manufacturing. And if they released to manufacturing without having a driver, then maybe they deserve what they get. Without software support, and FPGA design is just a pile of power-sucking metal. (And my partner, the EE, hates it when I raise that point.)

Peter wrote:

Please tell me this is some sort of embedded system… where you can fool-around with stuff and not be concerned about other things in the configuration changing.

I did enough screwy drivers for embedded, but I would run away from this one. Bottom line is the FPGA designer produced a piece of garbage.

I’m going to agree with everyone on here …

a) PCIe is PCIe; it’s not very hard to get good IP in FPGA libraries these days, some of it even for free (I use the Numato Tagus for PCIe FPGA development [ https://numato.com/product/tagus-artix-7-pci-express-development-board/ ] and for that entry level FPGA Vivado has PCIe 2.0 IP for free. There’s absolutely no excuse for not building the FPGA the right way … not some bizarre “bus with a wire” design

b) It’s good that you have a GPIO exposed on the FPGA, but not for what you might think … I always design in a GPIO that can be toggled with a BAR register for “blinky LED” sorts of things from the driver (a “blinky LED” is the hardware equivalent of “Hello World” for software) … but it’s only for that, never for controlling something and definitely never for IO’s (and no, the PCIe bus won’t have a clue what to do with a level interrupt, that will go into a different bus altogether)

c) @“Peter_Viscarola_(OSR)” is also correct, KMDF will simplify all of the interrupt stuff in a very easy WDFInterrupt object (a few more lines for MSI interrupts), you don’t need any of that WDM stuff in there. There are good samples on GitHub of this as well as discussions in prior years on this site about hooking this up

d) Without question your hardware design is irretrievably broken. If it were me I would communicate gently but firmly to your manager that the EE should be strongly encouraged to pursue other career options outside of anything relating to hardware, see if you can connect a JTAG programmer to it somewhere, bulk erase the whole thing and start from scratch; again, there are good PCIe libraries out there. If they insist you carry on then tell them it will take not weeks, not months but years of work and you’re more than happy to disappear into a cave until 2026

e) @“Peter_Viscarola_(OSR)” Yep, a PCIe analyzer is really high up on my wish list but until that Powerball comes in (or a good analyzer is less than the cost of an house in most of the country) it’s just me and the “blinky LED” … :slight_smile:

Come now. That is NEVER literally true.

Your FPGA team has produced a fatally flawed design

It might not be “his” FPGA… They might have an FPGA they sourced from somewhere out of house, that they’re building onto a board.

OK… so, I agree, that’s rarely true. But, it happens. I had a client who had an FPGA that did some processing on particular device, PLUS a (reasonably expensive) PLX local bus to PCIe bridge chip. Government contract, of course. :wink:

Peter

Thank you for your comments, like Peter said it is not internal design. Redesign is other big topic which i dont want to introduce here to not lose main goal.

As a proof of concept for my managment I have modified FPGA design and rised PCIe legacy line interrupt there. My driver imediatelly discovered it.
In the ISR routine I have disabled interrupt, acknowledged it (PCIe FPGA memory) and returned TRUE to the framework.

Now, the thing is that ISR is called again and agin. I pretty sure that it is not from other device - until I will rise interrupt from PCIe FPGA this ISR is never called. Do I also need to inform framework that i managed with interrupt in other way than returning TRUE value ? Is it possible to get directly source of interrupt to verify it ? At this moment I am using example where interrupt is verified in ISR by reading status from device memory.

As ISR is called all the time my OS is almost hanging :confused:

Hm, I don’t know the details, but to me it sounds like even if you acknowledge the IRQ, you don’t clear the condition (in the FPGA) that causes the IRQ, so another IRQ is immediately generated after the previous one was handled. I.e. like a level-triggered IRQ, where the level is not cleared when the IRQ is handled.

Well… traditionally, for Line Based Interrupts (which are level triggered), your ISR needs to tell the device to stop asserting the interrupt. You also need a register bit that indicates that it was your device that requested the interrupt, so you know whether to return TRUE or FALSE.

Until your device stops interrupting, you’l keep getting interrupts. If you’re returning TRUE and your ISR keeps getting called, that interrupt has remained asserted.

Is it possible to get directly source of interrupt to verify it ?

Well, that’s what you do when you check the “interrupt request” bit set by your device when it asserts the interrupt request line. The OS doesn’t know who’s asserting that line or indeed how many devices are asserting it.

I’ll bet your device is still asserting the interrupt line.

Of course this is Express, so all this “assert” and “line” stuff is virtual and emulated.

Peter

I have set all this things in the device. Interrupt has been acknowledged and interrupt enable flag has been also desactivated.

It sounds that it is more connected to the device than Windows driver so I switch to FPGA side and debug it.

Thanks a lot for the help, that’s let me to diagnose problem and I have hope finally i will finish with good design.

Problem solved.

e) @“Peter_Viscarola_(OSR)” Yep, a PCIe analyzer is really high up on my wish list but until that Powerball comes in (or a good analyzer is less than the cost of an house in most of the country) it’s just me and the “blinky LED” … :slight_smile:

Actually used one is not so expensive depending on the requirements. Bigger cost will be additional proper cabling.

https://www.ebay.com/itm/233031634795?hash=item3641c48f6b:g:ld0AAOSwnG1cBW~E

LOL… yeah, good luck with a piece of complex, expensive, equipment from eBay. And… which version of PCIe will that support?

And, yes…. Still very expensive when you price out the cost of the necessary Interposer.

When I last checked, I couldn’t get a good PCIe V3 x4 (even) analyzer and “PCIe riser slot” type interposer for less than US$100K. And then… consider that if you had a device to debug (like and FPGA) that was soldered down on the main board and not on an add-in card, you’d need an additional Interposer for that (and the soldering skills to attach it without toasting the board). Beyond us here at OSR, for sure.

Peter

PCIe level TLP debugging is way beyond my abilities to diagnose, fix (or even comprehend …) … I depend on the FPGA IP’s in various libraries to get things right, if there are problems I’ll just switch to a different IP …

PCIe level TLP debugging is way beyond my abilities to diagnose, fix (or even comprehend …)

You say that, I bet, never having tried it.

Having lived for a few months with a PCIe bus analyzer, I can tell you… even as a humble software engineer… knowing exactly which registers are accessed when, and when each interrupt is generated by a device and how… is awesome.

Rant follows:

I’ve often said that one of the worst things to happen in hardware design is that with FPGAs everybody thinks they can build their own hardware. It lead to people who have no clue about computer architecture in general or Express in particular building devices. And this lack of knowledge invariably leads hardware that ranges from sad and sub-optimal to patently ridiculous. I guess one of the good things about this trend is that it allows us to offer more services to those who recognize the need. The problem is, most device builders think their IP will save them, and thus don’t see the need. Those PCIe DMA IPs take lots of parameters.

Not sayin this is you Mr. @craig_howard …. Totally not saying that at all. I’m thinking about my experiences over the many years. Soooo many truly smart client engineers who don’t know what they don’t know… but who want to argue over adding specific device features… and use, like, 32 MSIs when one will do.

Peter

1 Like

Those who ‘don’t know what they don’t know’ are all too common. learning hard things takes effort even for smart people