I’m inexperienced in PCIe and driver writing so I need some help.
We have a PCIe device (using an NXP LS1046A if that is of interest) and we can communicate with it using shared memory mapped in a BAR. However that is just with polling on both sides. Now we’d like to make use of interrupts (MSI) to not have to poll memory continuously.
I’ve read about WdfCreateInterrupt/Isr/Dpc and so on but to me that only seems to handle interrupts coming from the device (which I’d need to implement as well of course). But I couldn’t find anything about sending an MSI to a device. In the device’s PCI config area is an MSI_Address and also MSI_Data. I believe the data has to be written to the address to trigger an MSI. Do I have to do this myself or does the PCIe system do it? If so what function do I need to call to trigger it? If this is done using an interrupt object: Can the same be used for sending and handling interrupts? If I need to do it myself: Does this address have to be mapped in a BAR as well?
I’ve looked through the Microsoft samples on Github but didn’t see (or quite possibly overlooked) what I need to do. If this is already used in one of these samples you can point me there.
Interrupts go in exactly one direction – from the hardware to the processor. You cannot “trigger” an MSI. If you need to talk to your device, then you just talk to your device. There’s no POINT for such an interrupt. The device is spending all of its time just waiting for something to do. The processor is busy doing other things. That’s why we need interrupts.
This is not the case for our device, it will control systems like machines (with a real time OS) and we will make use of all cores in the device to be able to handle all the load. Polling for communication will unnecessarily add more load. I think I can trigger an interrupt by directly writing into a register in the device which also needs to be mapped in a BAR.
I assumed that MSI or PCIe would allow to send messages in both directions and therefore the driver could also notify the endpoint device that new data is available without it needing to poll for it. But I guess that was wrong and explains why I couldn’t find information about it. Thanks for clearing it up.
So you have an interrupt mechanism for the device, implemented as a bar register. That is all you need. An MSI based device interrupt (not that it exists) would not be any particular advantage. The advantages of MSI (not shared, specificity reduces the need for additional read operations, etc) are not relevant.
Microsoft describes “Using an Interrupt to Wake a Device” so in some way it IS possible to send an interrupt the other way. And in PCIe descriptions it is also said that MSI communication is two-ways. If that was possible then I could have implemented it in a generic way, no matter what SOC we’re using. But I’ll stay with the register access for now, even if this is hardware dependant. Now on to the other direction…
There is no facility to generate MSI Host to device interrupts. Your hardware should have register based interrupts defined for host to device signaling.
Skipping many complexities, the CPU is just one processor in a system. Many (most / all?) peripheral cards have processors too. Each processor has a particular execution context, and a stream of instructions. If those instructions are a program to evaluate the first billion digits of Pi, then that could take a long time to complete. But what if another component needs attention while this calculation is ongoing? That’s where interrupts come in - stop what you are doing and pay attention to me!
Most ‘common’ interrupts are initiated by a device and indicate that that device needs attention from the CPU and the OS. The device raises a level (level triggered), changes and edge (edge triggered) or sends a message (message signal) onto the bus, the bus and chipset do their thing, and eventually the CPU notices that the device wants attention and does something about it.
But the reverse communication is possible also - the CPU will ‘send’ an interrupt to the device using some mechanism that the processor on the device supports.
I am skipping many details, but maybe this high level description will help you to understand the general arrangement. It should be noted that many devices, especially high performance devices - user hybrid polling or straight polling to improve performance. This is a complex subject and as with all things performance related, your milage will vary
I have another question: Is it allowed that a device can send both INTx and MSI (without reconfiguring)? Does it matter for the driver (driver-side) how the interrupt was delivered or is only one kind set up? I can see in the device’s registers that MSI was enabled but INTx was disabled. Probably after seeing the MSI capability the RC disabled INTx. Would it still work if I enabled it again and sent a INTx or does this have to be done before the device is scanned?
Thanks
In general, interrupts are not things that come an go independently of the devices that can signal them. Drivers are handed resources and are expected to connect those interrupts during hardware prepare or device add. I don’t know that there is any rule why a device can’t have multiple kinds simultaneously, but the obvious question is why would you? The more basic interrupt types exist to maintain compatibility with older hardware or save money by reducing the complexity of the circuits (an argument that is very flimsy in 2023). If your hardware supports MSI, why would you want to trigger interrupts in any other way?
I only asked because I’m still trying to make sending interrupts work and I tried both variants but none worked so far. If the INTx was intentionally disabled because MSIs are expected then that would explain why this didn’t work. So I have to focus on making MSIs work.