WDF never calls ISR callback

I found strange issue and cannot find a root of problem.
I used 16 MSI for my device in a WDF driver, It worked without any problems, I registered required MSIs in INF etc (actually, the driver worked on any Windows 10 that I have)

After some refactoring, I decreased MSI to 8 (the device side also uses only 8 MSI), so I also changed INF file - after that I got a problems.
Windows always gives me all required MSI (8 or 16) and I register them with WdfInterruptCreate

So, I have an issue on some PCs, the problem is reproduced only when I try to use 8 MSI:
WDF and driver work without any problems, device also has no problems but my ISR callback is never called.
I did not find any errors in WDF or system logs, I checked hardware conflicts etc.
But a device works with DMA memory without problem (I have true data in DMA) but I have no any interrupts from device.

If I change only INF file (change 8 MSI to 16 MSI) and install the INF again (the device and sys-file are the same) - Windows allocates 16 MSI and driver works ok, WDF calls my ISR callback (really a device uses only 8 MSI from the 16 allocated)
Also, I have a few PC that work ok with both 8 and 16 MSI.

I don’t know how to find the reason of such behavior or ever how to debug the issue,
please give me some advises.

I would like to check that a device really sends MSI, but how can I do that without ISR callback?

I found that MSI from my device are delivered to another driver not to my driver.

This secondary driver has self KINTERRUPT structures and it uses 16 MSI according to msinfo32.exe.
This secondary driver also uses WDF and it has self WDFINTERRUPT.
This secondary device is Intel controller.

There are no conflicts for this two devices, because we use MSI and also I checked all KINTERRUPT and WDFINTERRUPT objects for both devices - they are use different interrupt.Vectors values.

Why I know, that those interrupts from my device? - it’s because my device generates about 200 interrupts per second and I printed statistics (in Windbg) about interrupts for this secondary device and I can say - it handles my interrupts (the same statistic values as for my device in case my device work ok) . Also, usually this secondary device doesn’t handle interrupts at all (I mean a Isr callback of secondary device usually does not execute if my device in IDLE mode)

As I know, MSI are not shared interrupts, so the driver for secondary device cannot get my MSIs at all.

I tried disable this secondary device - it did not help me.

Any idea why is it happen?

I checked all interrupt’s logic from the KiInterruptDispatch to WDF callback but I still cannot find the reason.

The logs for case with 16 MSI (this solution is working ok):

  1. Interrupt resources
    Translated: Type=2 ShareDisposition=1 Flags=3 Level=4 Vector=0x40 Affinity=0xff
    Raw: Type=2 ShareDisposition=1 Flags=3 MessageCount=0x10 Vector=0xFFFFFFA9 Affinity=0

  2. Device’s PCI configuration space:
    50: CapID 05 MSI Capability
    51: NextPtr 70
    52: MsgCtrl 64BitCapable MSIEnable MultipleMsgEnable:4 (0x10) MultipleMsgCapable:5 (0x20)
    54: MsgAddr feeff00c
    58: MsgAddrHi 0
    5c: MsData 4940

where MsData 0x4940 (bin 0100 1001 0100 0000) is (according to Data format from the PCI specification):
15th bit: Trigger Mode (0=Edge)
14th bit: Trigger Level (1=Assert)
13th-12th bits: reserved
11th-8th bits: Delivery mode (001=Lowest Priority)
7th-0th bits: Vector (0100 0000 = 0x40) - the value is correct

  1. !acpiirqarb:
    0000000000000040 - 000000000000004f D ffffb1818459f800 (MyDriver) A:ffffd80f0150c210 IRQ(GSIV):ffffffa9
    according to acpi arbiter IRQ (ffffffa9) is valid and the vector range is valid too (0x40-0x4f - vectors for a 16 MSI numbers)

  2. Breakpoint to KiInterruptDispatch :
    as I understood, it is the first function in the Windows kernel that handles interrupts,
    in my varsion of Windows a RSI register is a pointer to a _KINTERRUPT structure
    my condition breakpoint to KiInterruptDispatch breaks only if _KINTERRUPT::IRQ value is 4
    after validation about 100 breaks I saw, it works ok, _KINTERRUPT describes WDF callback and WDF!FxInterrupt for my ISR callback (WDF delivers MSI to my driver)

All works as expected.

The logs for case with 8 MSI (this case is not working for me, all changes are only in INF file: 16 => 8):

  1. Interrupt resources
    InterruptTranslated: Type=2 ShareDisposition=1 Flags=3 Level=0xb Vector=0xB8 Affinity=0xff
    Raw: Type=2 ShareDisposition=1 Flags=3 MessageCount=8 Vector=0xFFFFFFA9 Affinity 0

  2. Device’s PCI configuration space:
    50: CapID 05 MSI Capability
    51: NextPtr 70
    52: MsgCtrl 64BitCapable MSIEnable MultipleMsgEnable:3 (0x8) MultipleMsgCapable:5 (0x20)
    54: MsgAddr feeff00c
    58: MsgAddrHi 0
    5c: MsData 49b8

where MsData 0x49B8 (bin 0100 1001 1011 1000) is (according to Data format from the PCI specification):
15th bit: Trigger Mode (0=Edge)
14th bit: Trigger Level (1=Assert)
13th-12th bits: reserved
11th-8th bits: Delivery mode (001=Lowest Priority)
7th-0th bits: Vector (1011 1000 = 0xb8) - the value is correct, only Vector value was changed

  1. !acpiirqarb:
    00000000000000a5 - 00000000000000a5 D ffffb1817b52b060 (nhi) A:ffffd80f0877b8d0 IRQ(GSIV):ffffffab - really it is Intel’s driver
    00000000000000b8 - 00000000000000bf D ffffb18181915800 (MyDriver) A:ffffd80f070d0e40 IRQ(GSIV):ffffffa9

according to acpi arbiter IRQ (ffffffa9) is valid and the vector range is valid too (0xb8-0xbf - vectors for a 8 MSI numbers)

  1. Breakpoint to KiInterruptDispatch :
    In this configuration (8 MSI) KiInterruptDispatch has never stopped with _KIINTERUPT for MyDriver
    When my device generates MSI for my driver, interrupt controller always calls KiInterruptDispatch with wrong _KINTERRUPT!
    I mean, that interrupts from my device are really deliver to another driver because KiInterruptDispatch always uses _KINTERRUPT for Intel’s driver but it should use my _KINTERRUPT

nt!_KINTERRUPT
+0x000 Type : 0n22
+0x002 Size : 0n256
+0x008 InterruptListEntry : _LIST_ENTRY [0x0000000000000000 - 0x0000000000000000]
+0x018 ServiceRoutine : 0xfffff80550412d30 unsigned char Wdf01000!FxInterrupt::_InterruptThunk+0 +0x020 MessageServiceRoutine : (null) +0x028 MessageIndex : 0 +0x030 ServiceContext : 0xffffb18183b29bc0 Void
+0x038 SpinLock : 0
+0x040 TickCount : 0
+0x048 ActualLock : 0xffffb1816c65f090 -\> 0 +0x050 DispatchAddress : 0xfffff800b7e34c00 void nt!KiInterruptDispatch+0
+0x058 Vector : 0xa5
+0x05c Irql : 0xa ‘’
+0x05d SynchronizeIrql : 0xb ‘’
+0x05e FloatingSave : 0 ‘’
+0x05f Connected : 0x1 ‘’
+0x060 Number : 0
+0x064 ShareVector : 0 ‘’
+0x065 EmulateActiveBoth : 0 ‘’
+0x066 ActiveCount : 0
+0x068 InternalState : 0n4
+0x06c Mode : 1 ( Latched )
+0x070 Polarity : 0 ( InterruptPolarityUnknown )
+0x074 ServiceCount : 0
+0x078 DispatchCount : 0
+0x080 PassiveEvent : (null)
+0x088 TrapFrame : 0xfffff800bac709d0 _KTRAP_FRAME +0x090 DisconnectData : (null) +0x098 ServiceThread : (null) +0x0a0 ConnectionData : 0xffffb18180a9b640 _INTERRUPT_CONNECTION_DATA
+0x0a8 IntTrackEntry : 0xffffb1818217e010 Void +0x0b0 IsrDpcStats : _ISRDPCSTATS +0x0f0 RedirectObject : 0xffffb1817b4bda50 Void
+0x0f8 Padding : [8] “”

For example, my device generates MSI #7, according to the information above, device uses Irql=0xa and Vector=0xBF for this MSI
but interrupt controller interpretes this message as Irql=0xa and Vector=0xa5 and calls KiInterruptDispatch with wrong _KINTERRUPT.

Questions:

  1. Both devices (my device and Intel device) use MSI and one IRQL (0xa) - I don’t think, that it is a problem, because IRQL is shareable, but anyway, how can I change IRQL value for my interrupts just to check that it is not a problem?

  2. How can I look a raw MSI data that device wrote to send MSI? I would like to check that MSI data from the device is valid

  3. Any idea, how is it possible? Why does interrupt controller uses wrong _KINTERRUPT ? How can I debug interrupt controller logic that handles interrupts’ messages?

Thanks

Your device must honor “Multiple Message Enable” field in Message Control register of MSI descriptor, and only send up to the number of vectors set in that field by the OS. If you request 8 vectors through INF, and the OS sets b’011 in “Multiple Message Enable” field, the device must not send messages 8 and above.

Alex, thank you for reply.
I will validate the device firmware.

Do you know a way, how can I get a raw messages data using Windbg ?
Can I validate MSI vector’s values that device uses for MSI from Windbg side?
e.g. debug interrupt controller or change MsgAddr address to my value etc

> How can I look a raw MSI data that device wrote to send MSI

To see the actual interrupt transaction, I’m afraid you need a bus analyzer.

Peter
OSR
@OSRDrivers

Solution:
It is proved as a windows limitation for specific hardware vendor
e.g.: windows doesn’t fully support all APICs