Thank you very much for your answer. Yes, I have only line interrupt in my device and I am not registering any MSI.
I moved interrupt object creation to the EvtDriverDeviceAdd but it didnāt solve my problem yet. I have enabled all additional logs (thanks for the suggestion).
Only one thing connected to the interrupt which I found is this line:
29: FxInterrupt::AssignResources - Is MSI? 0, MSI-ID 0, AffinityPolicy WdfIrqPolicyOneCloseProcessor, Priority WdfIrqPriorityUndefined, Group 0, Affinity 0x3, Irql 0x8, Vector 0x82
I was trying also to set up policy and priority but it didnāt change anything in calling my ISR.
btw. am I able to check somehow if CPU see my interrupt from hardware ?
I was trying also to set up policy and priority but it didnāt change anything in calling my ISR.
Donāt do ANY of that shit. Take out anything and everything related to your management of interrupts, except for your EvtDeviceInterruptEnable/Disable callbacks, your Dpc and ISR callbacks, EvtDevicePrepareHardware (where you now do nothing at all regarding your interrupts), and the creation of the WdfInterruptObject in EvtDriverDeviceAdd.
That logging output looks like things are all goodā¦
am I able to check somehow if CPU see my interrupt
No. I meanā¦ not without a PCI Bus Analyzer, which I donāt think was the intent of your question
SOMEbody needs to make an affordable PCI Bus Analyzerā¦ Iāve wanted one myself multiple times in the last two weeks.
In your initial post, you wrote this:
with oscilloscope interrupt line - interrupt is generated correctly by the hardware
This is a BIT confusingā¦ because you say youāre on an Express bus. And in that case, there IS no interrupt line. So, Iām a bit confused.
Maybe he means the source interrupt signal before the PCIe message generation logic.
One doesnāt need to buy a whole cow for a glass of milk.
Weāve rented a LeCroy analyzer some time ago; it was very expensive but came with set-up assistance of their engineer.
The guy was extremely helpful, not only he saved us time to connect the analyzer, he also looked at the data and immediately spotted some bugs.
So it was well worth the price.
This is a BIT confusingā¦ because you say youāre on an Express bus. And in that case, there IS no interrupt line. So, Iām a bit confused.
Thanks for the suggestion, I have checked what our hardware engineers done. They connected from FPGA to CPU PCIe bus but interrupt implemented on separate GPIO line. Now i need to manage with this type of setup.
I could implement second driver for GPIO controller and pass info from interrupt to my driver but it will be no more interrupt based signal. Additional delays etc.
Second option which I see is to try to conect to interrupt controller directly from my PCIe device driver and configure APIC to work with interrupt from interrupt capable GPIO. Here I am not sure hot to do proper connection to the APIC from my driver.
Neither of those options sound very reasonable to me. I donāt think I could make either one of them work in a way thatās architecturally sound, personally.
You have an FPGAā¦. Does it not include sufficient logic to send/receive data from the PCIe bus? If it does not, Iām not sure what to tell youā¦. If it does, you have everything you need to generate an MSI.
because of different reasons FPGA firmware is untouchable
And the board design is fixed? If so, you have what is effectively an unsupportable device design. Or, to be more clear, a device design that will not support interrupts. Generating GPIO interrupts from an Express device isā¦ just plain silly. Iām not even sure if, technically, you can make that work on Windows. I donāt expect the PCI bus driver would understand GPIO interrupts. And I donāt expect any of ACPI (including the Resource Hub) to understand GPIO resources associated with a device that has PCIe resources.
Please tell me this is some sort of embedded systemā¦ where you can fool-around with stuff and not be concerned about other things in the configuration changing.
Come now. That is NEVER literally true. Every hardware engineer knows their designs are not flawless.
What may be true is that you, as the software engineer, are afraid to raise an objection at this point. However, that is your job. Gather your evidence and make a PowerPoint. Your FPGA team has produced a fatally flawed design that cannot be made to work reliably in Windows. It needs to be fixed. Better now then after it has been released to manufacturing. And if they released to manufacturing without having a driver, then maybe they deserve what they get. Without software support, and FPGA design is just a pile of power-sucking metal. (And my partner, the EE, hates it when I raise that point.)
Please tell me this is some sort of embedded systemā¦ where you can fool-around with stuff and not be concerned about other things in the configuration changing.
I did enough screwy drivers for embedded, but I would run away from this one. Bottom line is the FPGA designer produced a piece of garbage.
a) PCIe is PCIe; itās not very hard to get good IP in FPGA libraries these days, some of it even for free (I use the Numato Tagus for PCIe FPGA development [ https://numato.com/product/tagus-artix-7-pci-express-development-board/ ] and for that entry level FPGA Vivado has PCIe 2.0 IP for free. Thereās absolutely no excuse for not building the FPGA the right way ā¦ not some bizarre ābus with a wireā design
b) Itās good that you have a GPIO exposed on the FPGA, but not for what you might think ā¦ I always design in a GPIO that can be toggled with a BAR register for āblinky LEDā sorts of things from the driver (a āblinky LEDā is the hardware equivalent of āHello Worldā for software) ā¦ but itās only for that, never for controlling something and definitely never for IOās (and no, the PCIe bus wonāt have a clue what to do with a level interrupt, that will go into a different bus altogether)
c) @āPeter_Viscarola_(OSR)ā is also correct, KMDF will simplify all of the interrupt stuff in a very easy WDFInterrupt object (a few more lines for MSI interrupts), you donāt need any of that WDM stuff in there. There are good samples on GitHub of this as well as discussions in prior years on this site about hooking this up
d) Without question your hardware design is irretrievably broken. If it were me I would communicate gently but firmly to your manager that the EE should be strongly encouraged to pursue other career options outside of anything relating to hardware, see if you can connect a JTAG programmer to it somewhere, bulk erase the whole thing and start from scratch; again, there are good PCIe libraries out there. If they insist you carry on then tell them it will take not weeks, not months but years of work and youāre more than happy to disappear into a cave until 2026
e) @āPeter_Viscarola_(OSR)ā Yep, a PCIe analyzer is really high up on my wish list but until that Powerball comes in (or a good analyzer is less than the cost of an house in most of the country) itās just me and the āblinky LEDā ā¦
Come now. That is NEVER literally true.
ā¦
Your FPGA team has produced a fatally flawed design
It might not be āhisā FPGAā¦ They might have an FPGA they sourced from somewhere out of house, that theyāre building onto a board.
OKā¦ so, I agree, thatās rarely true. But, it happens. I had a client who had an FPGA that did some processing on particular device, PLUS a (reasonably expensive) PLX local bus to PCIe bridge chip. Government contract, of course.
Thank you for your comments, like Peter said it is not internal design. Redesign is other big topic which i dont want to introduce here to not lose main goal.
As a proof of concept for my managment I have modified FPGA design and rised PCIe legacy line interrupt there. My driver imediatelly discovered it.
In the ISR routine I have disabled interrupt, acknowledged it (PCIe FPGA memory) and returned TRUE to the framework.
Now, the thing is that ISR is called again and agin. I pretty sure that it is not from other device - until I will rise interrupt from PCIe FPGA this ISR is never called. Do I also need to inform framework that i managed with interrupt in other way than returning TRUE value ? Is it possible to get directly source of interrupt to verify it ? At this moment I am using example where interrupt is verified in ISR by reading status from device memory.
As ISR is called all the time my OS is almost hanging
Hm, I donāt know the details, but to me it sounds like even if you acknowledge the IRQ, you donāt clear the condition (in the FPGA) that causes the IRQ, so another IRQ is immediately generated after the previous one was handled. I.e. like a level-triggered IRQ, where the level is not cleared when the IRQ is handled.
Wellā¦ traditionally, for Line Based Interrupts (which are level triggered), your ISR needs to tell the device to stop asserting the interrupt. You also need a register bit that indicates that it was your device that requested the interrupt, so you know whether to return TRUE or FALSE.
Until your device stops interrupting, youāl keep getting interrupts. If youāre returning TRUE and your ISR keeps getting called, that interrupt has remained asserted.
Is it possible to get directly source of interrupt to verify it ?
Well, thatās what you do when you check the āinterrupt requestā bit set by your device when it asserts the interrupt request line. The OS doesnāt know whoās asserting that line or indeed how many devices are asserting it.
Iāll bet your device is still asserting the interrupt line.
Of course this is Express, so all this āassertā and ālineā stuff is virtual and emulated.
e) @āPeter_Viscarola_(OSR)ā Yep, a PCIe analyzer is really high up on my wish list but until that Powerball comes in (or a good analyzer is less than the cost of an house in most of the country) itās just me and the āblinky LEDā ā¦
Actually used one is not so expensive depending on the requirements. Bigger cost will be additional proper cabling.
LOLā¦ yeah, good luck with a piece of complex, expensive, equipment from eBay. Andā¦ which version of PCIe will that support?
And, yesā¦. Still very expensive when you price out the cost of the necessary Interposer.
When I last checked, I couldnāt get a good PCIe V3 x4 (even) analyzer and āPCIe riser slotā type interposer for less than US$100K. And thenā¦ consider that if you had a device to debug (like and FPGA) that was soldered down on the main board and not on an add-in card, youād need an additional Interposer for that (and the soldering skills to attach it without toasting the board). Beyond us here at OSR, for sure.
PCIe level TLP debugging is way beyond my abilities to diagnose, fix (or even comprehend ā¦) ā¦ I depend on the FPGA IPās in various libraries to get things right, if there are problems Iāll just switch to a different IP ā¦
PCIe level TLP debugging is way beyond my abilities to diagnose, fix (or even comprehend ā¦)
You say that, I bet, never having tried it.
Having lived for a few months with a PCIe bus analyzer, I can tell youā¦ even as a humble software engineerā¦ knowing exactly which registers are accessed when, and when each interrupt is generated by a device and howā¦ is awesome.
Rant follows:
Iāve often said that one of the worst things to happen in hardware design is that with FPGAs everybody thinks they can build their own hardware. It lead to people who have no clue about computer architecture in general or Express in particular building devices. And this lack of knowledge invariably leads hardware that ranges from sad and sub-optimal to patently ridiculous. I guess one of the good things about this trend is that it allows us to offer more services to those who recognize the need. The problem is, most device builders think their IP will save them, and thus donāt see the need. Those PCIe DMA IPs take lots of parameters.
Not sayin this is you Mr. @craig_howard ā¦. Totally not saying that at all. Iām thinking about my experiences over the many years. Soooo many truly smart client engineers who donāt know what they donāt knowā¦ but who want to argue over adding specific device featuresā¦ and use, like, 32 MSIs when one will do.