I found strange issue and cannot find a root of problem.
I used 16 MSI for my device in a WDF driver, It worked without any problems, I registered required MSIs in INF etc (actually, the driver worked on any Windows 10 that I have)
After some refactoring, I decreased MSI to 8 (the device side also uses only 8 MSI), so I also changed INF file - after that I got a problems.
Windows always gives me all required MSI (8 or 16) and I register them with WdfInterruptCreate
So, I have an issue on some PCs, the problem is reproduced only when I try to use 8 MSI:
WDF and driver work without any problems, device also has no problems but my ISR callback is never called.
I did not find any errors in WDF or system logs, I checked hardware conflicts etc.
But a device works with DMA memory without problem (I have true data in DMA) but I have no any interrupts from device.
If I change only INF file (change 8 MSI to 16 MSI) and install the INF again (the device and sys-file are the same) - Windows allocates 16 MSI and driver works ok, WDF calls my ISR callback (really a device uses only 8 MSI from the 16 allocated)
Also, I have a few PC that work ok with both 8 and 16 MSI.
I don’t know how to find the reason of such behavior or ever how to debug the issue,
please give me some advises.
I would like to check that a device really sends MSI, but how can I do that without ISR callback?
I found that MSI from my device are delivered to another driver not to my driver.
This secondary driver has self KINTERRUPT structures and it uses 16 MSI according to msinfo32.exe.
This secondary driver also uses WDF and it has self WDFINTERRUPT.
This secondary device is Intel controller.
There are no conflicts for this two devices, because we use MSI and also I checked all KINTERRUPT and WDFINTERRUPT objects for both devices - they are use different interrupt.Vectors values.
Why I know, that those interrupts from my device? - it’s because my device generates about 200 interrupts per second and I printed statistics (in Windbg) about interrupts for this secondary device and I can say - it handles my interrupts (the same statistic values as for my device in case my device work ok) . Also, usually this secondary device doesn’t handle interrupts at all (I mean a Isr callback of secondary device usually does not execute if my device in IDLE mode)
As I know, MSI are not shared interrupts, so the driver for secondary device cannot get my MSIs at all.
I tried disable this secondary device - it did not help me.
where MsData 0x4940 (bin 0100 1001 0100 0000) is (according to Data format from the PCI specification):
15th bit: Trigger Mode (0=Edge)
14th bit: Trigger Level (1=Assert)
13th-12th bits: reserved
11th-8th bits: Delivery mode (001=Lowest Priority)
7th-0th bits: Vector (0100 0000 = 0x40) - the value is correct
!acpiirqarb:
0000000000000040 - 000000000000004f D ffffb1818459f800 (MyDriver) A:ffffd80f0150c210 IRQ(GSIV):ffffffa9
according to acpi arbiter IRQ (ffffffa9) is valid and the vector range is valid too (0x40-0x4f - vectors for a 16 MSI numbers)
Breakpoint to KiInterruptDispatch :
as I understood, it is the first function in the Windows kernel that handles interrupts,
in my varsion of Windows a RSI register is a pointer to a _KINTERRUPT structure
my condition breakpoint to KiInterruptDispatch breaks only if _KINTERRUPT::IRQ value is 4
after validation about 100 breaks I saw, it works ok, _KINTERRUPT describes WDF callback and WDF!FxInterrupt for my ISR callback (WDF delivers MSI to my driver)
All works as expected.
The logs for case with 8 MSI (this case is not working for me, all changes are only in INF file: 16 => 8):
where MsData 0x49B8 (bin 0100 1001 1011 1000) is (according to Data format from the PCI specification):
15th bit: Trigger Mode (0=Edge)
14th bit: Trigger Level (1=Assert)
13th-12th bits: reserved
11th-8th bits: Delivery mode (001=Lowest Priority)
7th-0th bits: Vector (1011 1000 = 0xb8) - the value is correct, only Vector value was changed
!acpiirqarb:
00000000000000a5 - 00000000000000a5 D ffffb1817b52b060 (nhi) A:ffffd80f0877b8d0 IRQ(GSIV):ffffffab - really it is Intel’s driver
00000000000000b8 - 00000000000000bf D ffffb18181915800 (MyDriver) A:ffffd80f070d0e40 IRQ(GSIV):ffffffa9
according to acpi arbiter IRQ (ffffffa9) is valid and the vector range is valid too (0xb8-0xbf - vectors for a 8 MSI numbers)
Breakpoint to KiInterruptDispatch :
In this configuration (8 MSI) KiInterruptDispatch has never stopped with _KIINTERUPT for MyDriver
When my device generates MSI for my driver, interrupt controller always calls KiInterruptDispatch with wrong _KINTERRUPT!
I mean, that interrupts from my device are really deliver to another driver because KiInterruptDispatch always uses _KINTERRUPT for Intel’s driver but it should use my _KINTERRUPT
For example, my device generates MSI #7, according to the information above, device uses Irql=0xa and Vector=0xBF for this MSI
but interrupt controller interpretes this message as Irql=0xa and Vector=0xa5 and calls KiInterruptDispatch with wrong _KINTERRUPT.
Questions:
Both devices (my device and Intel device) use MSI and one IRQL (0xa) - I don’t think, that it is a problem, because IRQL is shareable, but anyway, how can I change IRQL value for my interrupts just to check that it is not a problem?
How can I look a raw MSI data that device wrote to send MSI? I would like to check that MSI data from the device is valid
Any idea, how is it possible? Why does interrupt controller uses wrong _KINTERRUPT ? How can I debug interrupt controller logic that handles interrupts’ messages?
Your device must honor “Multiple Message Enable” field in Message Control register of MSI descriptor, and only send up to the number of vectors set in that field by the OS. If you request 8 vectors through INF, and the OS sets b’011 in “Multiple Message Enable” field, the device must not send messages 8 and above.
Alex, thank you for reply.
I will validate the device firmware.
Do you know a way, how can I get a raw messages data using Windbg ?
Can I validate MSI vector’s values that device uses for MSI from Windbg side?
e.g. debug interrupt controller or change MsgAddr address to my value etc