I have two FPGA PCIe cards which are built on the same IP but differ in port counts. The cards use MSI interrupts and each port has it’s own message. In this case one card has 4 ports and the other has 8 ports. Each card loads and functions fine on it’s own but when I try and use both cards in the system the second card never loads and device manager shows STATUS_DEVICE_POWER_FAILURE.
Looking at the WDF log I see the following failure (note this is from a different run than the logs below):
52: FxInterrupt::Connect - IoConnectInterrupt(Ex) Failed, SpinLock 0xFFFF830C55DD6AD0, Vector 0x93, IRQL 0x9, Synchronize IRQL 0x9, Mode 0x1, ShareVector False, ProcessorGroup 0, ProcessorEnableMask 0xfff, FloatingSave False, 0xc00000ef(STATUS_INVALID_PARAMETER_1)
I setup the interrupts in my EvtDevicePrepareHardware callback by calling the very simple function shown below with the resource descriptors:
_Use_decl_annotations_
NTSTATUS
InterruptInitialize(
WDFDEVICE Device,
PCM_PARTIAL_RESOURCE_DESCRIPTOR InterruptRaw,
PCM_PARTIAL_RESOURCE_DESCRIPTOR InterruptTranslated
)
{
WDF_INTERRUPT_CONFIG interruptConfig;
NTSTATUS status = STATUS_SUCCESS;
ULONG irq;
PAGED_CODE();
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INTERRUPT, "%!FUNC! Entry");
auto deviceContext = DeviceGetContext(Device);
if (CM_RESOURCE_INTERRUPT_MESSAGE & InterruptTranslated->Flags) {
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INTERRUPT,
"Message Interrupt level 0x%0x, Vector 0x%0x, MessageCount %u",
InterruptTranslated->u.MessageInterrupt.Translated.Level,
InterruptTranslated->u.MessageInterrupt.Translated.Vector,
InterruptTranslated->u.MessageInterrupt.Raw.MessageCount);
}
else {
TraceEvents(
TRACE_LEVEL_ERROR, TRACE_INTERRUPT, "Legacy Interrupts Not Supported!");
return STATUS_INVALID_DEVICE_REQUEST;
}
// First we need to determine how many interrupts we need
auto numberOfPorts = READ_REGISTER_ULONG(deviceContext->MemoryBar[1].Buffer + kPortsPresent) & 0x1F;
WDF_INTERRUPT_CONFIG_INIT(&interruptConfig, InterruptIsr, InterruptDpc);
interruptConfig.InterruptRaw = InterruptRaw;
interruptConfig.InterruptTranslated = InterruptTranslated;
for (irq = 0; irq < numberOfPorts; irq++) {
status = WdfInterruptCreate(Device, &interruptConfig, WDF_NO_OBJECT_ATTRIBUTES, &deviceContext->InterruptObject[irq]);
if (NT_SUCCESS(status)) {
WDF_INTERRUPT_INFO interruptInfo;
WDF_INTERRUPT_INFO_INIT(&interruptInfo);
// Log some info
WdfInterruptGetInfo(deviceContext->InterruptObject[irq], &interruptInfo);
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INTERRUPT, "Created interrupt(%u): Signaled(%c), Num(%u), Vect(0x%0x)",
irq, interruptInfo.MessageSignaled ? 'Y' : 'n', interruptInfo.MessageNumber, interruptInfo.Vector);
}
else {
TraceEvents(TRACE_LEVEL_ERROR, TRACE_INTERRUPT,
"WdfInterruptCreate failed with status %!STATUS!", status);
break;
}
}
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INTERRUPT, "Created %u MSI interrupts.", irq);
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INTERRUPT, "%!FUNC! Exit");
return status;
}
When I added the WdfInterruptGetInfo and additional logging I noticed something interesting. In a good case interruptInfo.Vector increments with every call:
00000012 driver-windows 4 5420 3 12 11\22\2021-08:07:31:842 InterruptInitialize Entry
00000013 driver-windows 4 5420 3 13 11\22\2021-08:07:31:842 Message Interrupt level 0x4, Vector 0x40, MessageCount 0
00000014 driver-windows 4 5420 3 14 11\22\2021-08:07:31:842 Created interrupt(0): Signaled(Y), Num(0), Vect(0x40)
00000015 driver-windows 4 5420 3 15 11\22\2021-08:07:31:842 Created interrupt(1): Signaled(Y), Num(1), Vect(0x41)
00000016 driver-windows 4 5420 3 16 11\22\2021-08:07:31:842 Created interrupt(2): Signaled(Y), Num(2), Vect(0x42)
00000017 driver-windows 4 5420 3 17 11\22\2021-08:07:31:842 Created interrupt(3): Signaled(Y), Num(3), Vect(0x43)
00000018 driver-windows 4 5420 3 18 11\22\2021-08:07:31:842 Created interrupt(4): Signaled(Y), Num(4), Vect(0x44)
00000019 driver-windows 4 5420 3 19 11\22\2021-08:07:31:842 Created interrupt(5): Signaled(Y), Num(5), Vect(0x45)
00000020 driver-windows 4 5420 3 20 11\22\2021-08:07:31:842 Created interrupt(6): Signaled(Y), Num(6), Vect(0x46)
00000021 driver-windows 4 5420 3 21 11\22\2021-08:07:31:842 Created interrupt(7): Signaled(Y), Num(7), Vect(0x47)
00000022 driver-windows 4 5420 3 22 11\22\2021-08:07:31:842 Created 8 MSI interrupts.
00000023 driver-windows 4 5420 3 23 11\22\2021-08:07:31:842 InterruptInitialize Exit
In the bad case it gets does not increment:
00000042 driver-windows 4 2656 0 42 11\22\2021-08:07:37:339 InterruptInitialize Entry
00000043 driver-windows 4 2656 0 43 11\22\2021-08:07:37:339 Message Interrupt level 0x6, Vector 0x63, MessageCount 0
00000044 driver-windows 4 2656 0 44 11\22\2021-08:07:37:339 Created interrupt(0): Signaled(Y), Num(0), Vect(0x63)
00000045 driver-windows 4 2656 0 45 11\22\2021-08:07:37:339 Created interrupt(1): Signaled(Y), Num(1), Vect(0x63)
00000046 driver-windows 4 2656 0 46 11\22\2021-08:07:37:339 Created interrupt(2): Signaled(Y), Num(2), Vect(0x63)
00000047 driver-windows 4 2656 0 47 11\22\2021-08:07:37:339 Created interrupt(3): Signaled(Y), Num(3), Vect(0x63)
00000048 driver-windows 4 2656 0 48 11\22\2021-08:07:37:339 Created 4 MSI interrupts.
00000049 driver-windows 4 2656 0 49 11\22\2021-08:07:37:339 InterruptInitialize Exit
The vector is supplied by Windows and simply passed through. I do not believe I should be manipulating it. I return success to the EvtDevicePrepareHardware and then it immediately turns around and calls my EvtDeviceReleaseHardware callback due to the IoConnectInterrupt failure.
Any thoughts what I am missing here? Any help is much appreciated.
-Tom