I spent three days studying network and forum topics but i was not able to solve my problem until know.
I am quite new in writing windows drivers, I am preparing KMDF driver for my device connected to the PCIe. I am able to read and write to the memory area of the device, read BARs data, information about interrupts etc.
I have prepared code in WDF to manage with interrupts (old line interrupt triggered by low level) raised by my device. I see in WinDbg that FPGAdrvEvtInterruptEnable and FPGAdrvEvtInterruptDisable functions are called by framework. I have problem that my InterruptServiceRoutine is not called by framework.
What I have checked:
with oscilloscope interrupt line - interrupt is generated correctly by the hardware
with WinDbg - my ISR looks correctly registered in the framework:
First, please verify: This is for a Line Based Interrupt, and your device has registered for JUST that single line based interrupt, correct? You’re not enabling MSI or MSI-X via registry entries, and there are no MSI or MSI-X Capabilities registered for your device? Is that all correct?
If that is correct, move your call to WdfInterruptCreate from EvtDevicePrepareHardware to EvtDriverDeviceAdd. Create one WDFINTERRUPT. You will not specify any resources here.
Assuming all the above is correct, WDF will connect your one Line Based Interrupt to the one WDF interrupt you’ve created.
Another thing to do is after EvtDevicePrepareHardware bus run, dump the log with !wdfkd.WDFLOGDUMP and read the output carefully. Look for any diagnostics related to interrupts (they’ll be in plain English). Make sure you’ve enabled WDF Verifier with Verbose logging, and are running with Windows Driver Verifier enabled.
Thank you very much for your answer. Yes, I have only line interrupt in my device and I am not registering any MSI.
I moved interrupt object creation to the EvtDriverDeviceAdd but it didn’t solve my problem yet. I have enabled all additional logs (thanks for the suggestion).
Only one thing connected to the interrupt which I found is this line:
29: FxInterrupt::AssignResources - Is MSI? 0, MSI-ID 0, AffinityPolicy WdfIrqPolicyOneCloseProcessor, Priority WdfIrqPriorityUndefined, Group 0, Affinity 0x3, Irql 0x8, Vector 0x82
I was trying also to set up policy and priority but it didn’t change anything in calling my ISR.
btw. am I able to check somehow if CPU see my interrupt from hardware ?
I was trying also to set up policy and priority but it didn’t change anything in calling my ISR.
Don’t do ANY of that shit. Take out anything and everything related to your management of interrupts, except for your EvtDeviceInterruptEnable/Disable callbacks, your Dpc and ISR callbacks, EvtDevicePrepareHardware (where you now do nothing at all regarding your interrupts), and the creation of the WdfInterruptObject in EvtDriverDeviceAdd.
That logging output looks like things are all good…
am I able to check somehow if CPU see my interrupt
No. I mean… not without a PCI Bus Analyzer, which I don’t think was the intent of your question
SOMEbody needs to make an affordable PCI Bus Analyzer… I’ve wanted one myself multiple times in the last two weeks.
In your initial post, you wrote this:
with oscilloscope interrupt line - interrupt is generated correctly by the hardware
This is a BIT confusing… because you say you’re on an Express bus. And in that case, there IS no interrupt line. So, I’m a bit confused.
Maybe he means the source interrupt signal before the PCIe message generation logic.
One doesn’t need to buy a whole cow for a glass of milk.
We’ve rented a LeCroy analyzer some time ago; it was very expensive but came with set-up assistance of their engineer.
The guy was extremely helpful, not only he saved us time to connect the analyzer, he also looked at the data and immediately spotted some bugs.
So it was well worth the price.
This is a BIT confusing… because you say you’re on an Express bus. And in that case, there IS no interrupt line. So, I’m a bit confused.
Thanks for the suggestion, I have checked what our hardware engineers done. They connected from FPGA to CPU PCIe bus but interrupt implemented on separate GPIO line. Now i need to manage with this type of setup.
I could implement second driver for GPIO controller and pass info from interrupt to my driver but it will be no more interrupt based signal. Additional delays etc.
Second option which I see is to try to conect to interrupt controller directly from my PCIe device driver and configure APIC to work with interrupt from interrupt capable GPIO. Here I am not sure hot to do proper connection to the APIC from my driver.
Neither of those options sound very reasonable to me. I don’t think I could make either one of them work in a way that’s architecturally sound, personally.
You have an FPGA…. Does it not include sufficient logic to send/receive data from the PCIe bus? If it does not, I’m not sure what to tell you…. If it does, you have everything you need to generate an MSI.
because of different reasons FPGA firmware is untouchable
And the board design is fixed? If so, you have what is effectively an unsupportable device design. Or, to be more clear, a device design that will not support interrupts. Generating GPIO interrupts from an Express device is… just plain silly. I’m not even sure if, technically, you can make that work on Windows. I don’t expect the PCI bus driver would understand GPIO interrupts. And I don’t expect any of ACPI (including the Resource Hub) to understand GPIO resources associated with a device that has PCIe resources.
Please tell me this is some sort of embedded system… where you can fool-around with stuff and not be concerned about other things in the configuration changing.
Come now. That is NEVER literally true. Every hardware engineer knows their designs are not flawless.
What may be true is that you, as the software engineer, are afraid to raise an objection at this point. However, that is your job. Gather your evidence and make a PowerPoint. Your FPGA team has produced a fatally flawed design that cannot be made to work reliably in Windows. It needs to be fixed. Better now then after it has been released to manufacturing. And if they released to manufacturing without having a driver, then maybe they deserve what they get. Without software support, and FPGA design is just a pile of power-sucking metal. (And my partner, the EE, hates it when I raise that point.)
Please tell me this is some sort of embedded system… where you can fool-around with stuff and not be concerned about other things in the configuration changing.
I did enough screwy drivers for embedded, but I would run away from this one. Bottom line is the FPGA designer produced a piece of garbage.
a) PCIe is PCIe; it’s not very hard to get good IP in FPGA libraries these days, some of it even for free (I use the Numato Tagus for PCIe FPGA development [ https://numato.com/product/tagus-artix-7-pci-express-development-board/ ] and for that entry level FPGA Vivado has PCIe 2.0 IP for free. There’s absolutely no excuse for not building the FPGA the right way … not some bizarre “bus with a wire” design
b) It’s good that you have a GPIO exposed on the FPGA, but not for what you might think … I always design in a GPIO that can be toggled with a BAR register for “blinky LED” sorts of things from the driver (a “blinky LED” is the hardware equivalent of “Hello World” for software) … but it’s only for that, never for controlling something and definitely never for IO’s (and no, the PCIe bus won’t have a clue what to do with a level interrupt, that will go into a different bus altogether)
c) @“Peter_Viscarola_(OSR)” is also correct, KMDF will simplify all of the interrupt stuff in a very easy WDFInterrupt object (a few more lines for MSI interrupts), you don’t need any of that WDM stuff in there. There are good samples on GitHub of this as well as discussions in prior years on this site about hooking this up
d) Without question your hardware design is irretrievably broken. If it were me I would communicate gently but firmly to your manager that the EE should be strongly encouraged to pursue other career options outside of anything relating to hardware, see if you can connect a JTAG programmer to it somewhere, bulk erase the whole thing and start from scratch; again, there are good PCIe libraries out there. If they insist you carry on then tell them it will take not weeks, not months but years of work and you’re more than happy to disappear into a cave until 2026
e) @“Peter_Viscarola_(OSR)” Yep, a PCIe analyzer is really high up on my wish list but until that Powerball comes in (or a good analyzer is less than the cost of an house in most of the country) it’s just me and the “blinky LED” …
Come now. That is NEVER literally true.
…
Your FPGA team has produced a fatally flawed design
It might not be “his” FPGA… They might have an FPGA they sourced from somewhere out of house, that they’re building onto a board.
OK… so, I agree, that’s rarely true. But, it happens. I had a client who had an FPGA that did some processing on particular device, PLUS a (reasonably expensive) PLX local bus to PCIe bridge chip. Government contract, of course.
Thank you for your comments, like Peter said it is not internal design. Redesign is other big topic which i dont want to introduce here to not lose main goal.
As a proof of concept for my managment I have modified FPGA design and rised PCIe legacy line interrupt there. My driver imediatelly discovered it.
In the ISR routine I have disabled interrupt, acknowledged it (PCIe FPGA memory) and returned TRUE to the framework.
Now, the thing is that ISR is called again and agin. I pretty sure that it is not from other device - until I will rise interrupt from PCIe FPGA this ISR is never called. Do I also need to inform framework that i managed with interrupt in other way than returning TRUE value ? Is it possible to get directly source of interrupt to verify it ? At this moment I am using example where interrupt is verified in ISR by reading status from device memory.
As ISR is called all the time my OS is almost hanging
Hm, I don’t know the details, but to me it sounds like even if you acknowledge the IRQ, you don’t clear the condition (in the FPGA) that causes the IRQ, so another IRQ is immediately generated after the previous one was handled. I.e. like a level-triggered IRQ, where the level is not cleared when the IRQ is handled.
Well… traditionally, for Line Based Interrupts (which are level triggered), your ISR needs to tell the device to stop asserting the interrupt. You also need a register bit that indicates that it was your device that requested the interrupt, so you know whether to return TRUE or FALSE.
Until your device stops interrupting, you’l keep getting interrupts. If you’re returning TRUE and your ISR keeps getting called, that interrupt has remained asserted.
Is it possible to get directly source of interrupt to verify it ?
Well, that’s what you do when you check the “interrupt request” bit set by your device when it asserts the interrupt request line. The OS doesn’t know who’s asserting that line or indeed how many devices are asserting it.
I’ll bet your device is still asserting the interrupt line.
Of course this is Express, so all this “assert” and “line” stuff is virtual and emulated.
e) @“Peter_Viscarola_(OSR)” Yep, a PCIe analyzer is really high up on my wish list but until that Powerball comes in (or a good analyzer is less than the cost of an house in most of the country) it’s just me and the “blinky LED” …
Actually used one is not so expensive depending on the requirements. Bigger cost will be additional proper cabling.
LOL… yeah, good luck with a piece of complex, expensive, equipment from eBay. And… which version of PCIe will that support?
And, yes…. Still very expensive when you price out the cost of the necessary Interposer.
When I last checked, I couldn’t get a good PCIe V3 x4 (even) analyzer and “PCIe riser slot” type interposer for less than US$100K. And then… consider that if you had a device to debug (like and FPGA) that was soldered down on the main board and not on an add-in card, you’d need an additional Interposer for that (and the soldering skills to attach it without toasting the board). Beyond us here at OSR, for sure.
PCIe level TLP debugging is way beyond my abilities to diagnose, fix (or even comprehend …) … I depend on the FPGA IP’s in various libraries to get things right, if there are problems I’ll just switch to a different IP …