Waterfall of interrupt events

NTDEV
1

When I create an interrupt object in my driver the interrupt service routine is called constantly. This starts already at boot time, when the driver is loaded.
Now I was wondering how I can best debug this; could this be a hardware failure, or am I doing something terribly wrong in my driver.
The device is a PCIe device (so MSI based interrupts).
In the ISR, I retrieve the interrupt info and get the following print out (1000’s a second):

Enter: MSI? No, Interrupt type: Level Triggered, Share disposition: Shared, MessageID: 0, Int Vector: 387

This is my code:
NTSTATUS EvtDriverDeviceAdd(__in WDFDRIVER Driver, __in PWDFDEVICE_INIT DeviceInit)
{
PAGED_CODE();

WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&attributes, DEVICE_EXTENSION);
attributes.EvtCleanupCallback = (PFN_WDF_OBJECT_CONTEXT_CLEANUP)EvtDeviceExtensionCleanup;
attributes.SynchronizationScope = WdfSynchronizationScopeDevice;

WDF_PNPPOWER_EVENT_CALLBACKS_INIT(&pnpPowerCallbacks);
pnpPowerCallbacks.EvtDevicePrepareHardware = EvtDevicePrepareHardware;
pnpPowerCallbacks.EvtDeviceReleaseHardware = EvtDeviceReleaseHardware;
WdfDeviceInitSetPnpPowerEventCallbacks(DeviceInit, &pnpPowerCallbacks);

status = WdfDeviceCreate(&DeviceInit, &attributes, &device);

WDF_INTERRUPT_CONFIG_INIT(&interruptConfig,EvtInterruptIsr,EvtInterruptDpc);
interruptConfig.EvtInterruptEnable = EvtInterruptEnable;
interruptConfig.EvtInterruptDisable = EvtInterruptDisable;
//interruptConfig.AutomaticSerialization = TRUE; /* Crashes when set to TRUE */

status = WdfInterruptCreate(device,&interruptConfig,WDF_NO_OBJECT_ATTRIBUTES,&pDevExt->Interrupt);
return status;
}

BOOLEAN VkEvtInterruptIsr(__in WDFINTERRUPT Interrupt, __in ULONG MessageID)
{
PVK_DEVICE_EXTENSION pDevExt;
BOOLEAN isRecognized = FALSE;
WDF_INTERRUPT_INFO Info;

WDF_INTERRUPT_INFO_INIT(&Info);
WdfInterruptGetInfo(Interrupt,&Info);

DbgPrint(“[Viking::VkEvtInterruptIsr] Enter: MSI? %s, Interrupt type: %s, Share disposition: %s, MessageID: %lu, Int Vector: %lu\n”,Info.MessageSignaled ? “Yes” : “No”, Info.Mode==Latched ? “Edge Triggered” : “Level Triggered”, Info.ShareDisposition == CmResourceShareShared ? “Shared” : “Exclusive”, MessageID, Info.Vector);

//WdfInterruptQueueDpcForIsr(pDevExt->Interrupt);
DbgPrint(“[Viking::VkEvtInterruptIsr] Leave\n”);

return isRecognized;
}

I read in the book Developing drivers with the Windows Driver Foundation that MSI interrupts are not supported prior to Vista (see page 538 at the top). Is this correct? And if so, how is XP handling MSI interrupts?
What does the Interrupt Vector number mean?
When I enable automatic serialization, the driver BSOD’s. What could be the reason for that?

Best regards,

Kurt

2

Lots of questions… Let me start by answering a couple of fundamentals.

Every MSI-capable device must also be capable of “falling back” to Line Base Interrupts (LBI, your standard IRQ mechanism). This is true because (a) Not all system hardware supports MSI, (b) Not all operating system versions support MSI. Prior to Vista, an MSI-capable card will therefore use ordinary line-based interrupts.

I suspect not having code that deals with this is the genesis of your problem.

I suggest you find and read the rather comprehensive document on WHDC about MSI (I wrote the original version).

Peter
OSR

3

Peter,

with PCIe there is no pin for line based interrupts.
I am just creating an interrupts object in the DddDevice eventhandler, and it starts calling the ISR.
So the basic question is: have I forgotten something?
Can those interrupts come from some other device? XP doesn’t support MSI, so maybe something is happening there under the hood that I didn’t catch?

/Kurt

4

There is this document from Microsoft about PCI, PCIe and PCIX:
http://download.microsoft.com/download/9/c/5/9c5b2167-8017-4bae-9fde-d599bac8184a/PCI-PCIe_FAQ.doc

5

On Fri, 3 Aug 2007 11:53:02 -0400 (EDT), xxxxx@barco.com wrote:

with PCIe there is no pin for line based interrupts. I am just
creating an interrupts object in the DddDevice eventhandler, and it
starts calling the ISR.
So the basic question is: have I forgotten something? Can those
interrupts come from some other device? XP doesn’t support MSI, so
maybe something is happening there under the hood that I didn’t
catch?

Is it not that the interrupt line (or whatever mechanism is connected) is shared
with another device. My ISR (for a PCIe device) is constantly called in most
systems (I’ve always assumed it was sharing with the network port, graphics
driver or other device). I check (and clear) the Interrupt Status Register for
my device and return TRUE if it was me (passing the Reason for Interrupt to the
DPC). If the status isn’t set it wasn’t my interrupt, I return FALSE.

If I stop and start the driver it seems I’m placed at the end of the queue (and
so I don’t get called unless the interrupt wasn’t claimed by an ISR further up
the queue).

I have to admit I’ve always found this way of handling the interrupt-stack a bit
weird. Surely it would be more efficient to call all the ISRs in the queue
regardless of whether earlier drivers “claimed” the interrupt or not - two
interrupts at once is entirely possible.

–
Graeme Griffiths
Image Processing Techniques Ltd.
+44 1256 345 927
*****************************************************************************
The contents of this Email and any files transmitted with it are confidential
and intended solely for the use of the individual or entity to whom it is
addressed.
The views stated herein do not necessarily represent the view of the company.
If you are not the intended recipient of this Email you may not copy, forward,
disclose or otherwise use it or any part of it in any form whatsoever.
If you have received this mail in error please Email the sender.
*****************************************************************************
Image Processing Techniques Ltd
Tel : +44(0)1256345910
Fax : +44(0)1256345911
Website : www.imageproc.com
Registered Office : Unit 3, Intec 2, Wade Road, Basingstoke, Hampshire, RG24 8NE
Registration Number : 3564291
Place of Registration : England
VAT Registration : 709198411

6

xxxxx@barco.com wrote:

with PCIe there is no pin for line based interrupts.

Well, there is no physical pin for this, since it is a serial bus, but
PCIe certainly does have the ability to send line-based interrupts. It
HAS to provide that, since it had to be able to look exactly like PCI
for the first couple of years of its life.

Message Interrupt Requester Transaction Packets are translated by the
root complex to look to the processor like a line-based interrupt.
Until relatively recently, most PCIe devices used this type of interrupt
exclusively. And as Doron said, even if you use MSI, your hardware has
to be able to fall back to this because not all platforms support MSI.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

7

Not sure what you mean. EVERY PCIe device is REQUIRED to support “legacy IntX” – You absolutely cannot be guaranteed that your PCIe device will be plugged into a system that has MSI. Or, even if you are, that you can use MSI on a system that supports it.

So, for ANY MSI DEVICE your device and your driver must be able to fall back to LBI.

I think that’s probably what you’re missing,

Peter
OSR

8

Well I didn’t know about the LBI. Thanks for that.
Then is it possible, like Graeme says, that my ISR is called for interrupts generated by other devices?

9

Yes unless you are on a seperate MSI interrupt you will be called shared.
If you return true from the ISR and it was not your interrupt you will get
the effect you describe. Also, if it is your device and you do not clear
the interrupt you will get this behavior.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
Remove StopSpam to reply

wrote in message news:xxxxx@ntdev…
> Well I didn’t know about the LBI. Thanks for that.
> Then is it possible, like Graeme says, that my ISR is called for
> interrupts generated by other devices?
>
>

10

> In the ISR, I retrieve the interrupt info and get the following print out (1000’s a second):

Enter: MSI? No, Interrupt type: Level Triggered, Share disposition: Shared,
MessageID: 0, Int Vector: 387

Please pay a special attention to the line ‘Int Vector: 387’. There are just 256 interrupt vectors in IDT, so that vector 387 (even in decimal notation) is just out of IDT limits. Therefore, there must be
some mistake here, i.e. all other interrupt info that you have shown us may be incorrect as well…

Anton Bassov

11

Graeme Griffiths wrote:

I have to admit I’ve always found this way of handling the interrupt-stack a bit
weird. Surely it would be more efficient to call all the ISRs in the queue
regardless of whether earlier drivers “claimed” the interrupt or not - two
interrupts at once is entirely possible.

I don’t think it would be more efficient. Multiple concurrent interrupts on the same line are possible, but rare. Then it is more efficient to pay a bit more overhead on the rare case, than a slightly smaller overhead in every single interrupt.

This is, of course, possibly for level-triggered interrupts only. For edge triggered ones the HAL does call *all* the connected handlers on each interrupt.

12

anton bassov wrote:

Please pay a special attention to the line ‘Int Vector: 387’. There are just 256
interrupt vectors in IDT, so that vector 387 (even in decimal notation) is
just out of IDT limits. Therefore, there must be some mistake here,

It is normal and (probably) not a mistake. Jake explained some time ago that the vector is actually an internal value, which is mapped to the actual IDT entry in different ways depending on the HAL.

13

> Please pay a special attention to the line ‘Int Vector: 387’. There are just 256

> interrupt vectors in IDT, so that vector 387 (even in decimal notation) is
> just out of IDT limits. Therefore, there must be some mistake here,

It is normal and (probably) not a mistake. Jake explained some time ago that the
vector is actually an internal value, which is mapped to the actual IDT entry in
different ways depending on the HAL.

Please note that mapping of IRQs to vectors (a.ka.a. translation of interrupt resources), on one hand, and *valid* vector number are totally different things…

Indeed, mapping of IRQs to vectors depends on HAL version. On PIC HAL vector= 0x30+ IRQ, so that IRQ0 maps to vector 0x30, IRQ1 maps to vector 0x31, etc, with interrupt priority implied by IRQ. On APIC HAL things are totally different - IOAPIC allows the system designers to map IRQs to vectors in any way they wish, with priority=vector number/15. This is why you cannot make any assumtion about vector number that will get assigned to your device. Although I haven’t seen his post, this is, apparently, what Jake explained…

However, no matter how how IRQ-to-interrupt mapping is done, vector number in itself still has to be valid, i.e fall within IDT limits. I think the best best example to demonstrate my logic is shuffling an array of N elements, which is originally filled with values in acsending order, i.e. like 1, 2, 3, 4, 5 …N, so that originally array ==x . Although after having shuffling this array you may find that array !=x, it has to be <= N, and this is not the case here. Therefore, I believe there must be some mistake…

Anton Bassov

14

anton bassov wrote:

>> Please pay a special attention to the line ‘Int Vector: 387’. There are just 256
>> interrupt vectors in IDT, so that vector 387 (even in decimal notation) is
>> just out of IDT limits. Therefore, there must be some mistake here,
>
> It is normal and (probably) not a mistake. Jake explained some time ago that
> the vector is actually an internal value, which is mapped to the actual IDT entry
> in different ways depending on the HAL.

Please note that mapping of IRQs to vectors (a.ka.a. translation of interrupt
resources), on one hand, and *valid* vector number are totally different
things…
However, no matter how how IRQ-to-interrupt mapping is done, vector number in
itself still has to be valid, i.e fall within IDT limits

No. I’m not talking about the relation between IRQ, priority and vector. I am talking about the relation between vector *as it appears in the interrupt object*, and the actual IDT entry.

The vector value in the interrupt object is for internal OS usage. Drivers are not supposed to change or interpret it. Then the OS is free to put whatever value or encoding it wants. See Jake’s description:

http://www.osronline.com/showThread.cfm?link=85997
http://www.osronline.com/showThread.cfm?link=79136

15

> I am talking about the relation between vector *as it appears in the interrupt

object*, and the actual IDT entry.

Oh, now I see…

Indeed, HAL adds an extra level of abstraction here (actually, this is exactly what the very term ‘HAL’ suggests, in the first place). For example, strictly speaking, you don’t “raise IRQL” on the CPU when raising task priority, simply because IRQL is, in actuality, just an index into the array that stores values to be written to the CPU’s Task Priority Register. When it comes to interrupt vectors, we are in exactly the same situation - for example, on the machine I am currently at ( XP SP2 that runs MP HAL) IRQ1 maps to IDT vector 0x93, but ‘Vector’ field of KINTERRUPT that corresponds to the kbd is 0x1b4, rather than 0x93. As a result of this abstraction, drivers can be built for different platofrms from the same sources, as long as these sources don’t try to “get under the hood”…

Anton Bassov

16

Please don’t assume that you understand what the value in “Vector” means.
It may be a raw IDT value or or it may be something that encodes information
beyond just the vector. If you want to know what IDT value your driver is
attached to, in the debugger, type:

!idt

  • Jake Oshins
    Windows Kernel Team
    (author of the code that can return values greater than 255 in Vector)

wrote in message news:xxxxx@ntdev…
>> In the ISR, I retrieve the interrupt info and get the following print out
>> (1000’s a second):
>
>> Enter: MSI? No, Interrupt type: Level Triggered, Share disposition:
>> Shared,
>> MessageID: 0, Int Vector: 387
>
>
> Please pay a special attention to the line ‘Int Vector: 387’. There are
> just 256 interrupt vectors in IDT, so that vector 387 (even in decimal
> notation) is just out of IDT limits. Therefore, there must be
> some mistake here, i.e. all other interrupt info that you have shown us
> may be incorrect as well…
>
> Anton Bassov
>
>

17

If two interrupts happen at once, the interrupt controller will assert again
as soon as IRQL is lowered after the first ISR clears its interrupting event
and returns TRUE from its ISR.

  • Jake Oshins

“Graeme Griffiths” wrote in message
news:xxxxx@ntdev…
> On Fri, 3 Aug 2007 11:53:02 -0400 (EDT), xxxxx@barco.com wrote:
>
>> with PCIe there is no pin for line based interrupts. I am just
>> creating an interrupts object in the DddDevice eventhandler, and it
>> starts calling the ISR.
>> So the basic question is: have I forgotten something? Can those
>> interrupts come from some other device? XP doesn’t support MSI, so
>> maybe something is happening there under the hood that I didn’t
>> catch?
>
> Is it not that the interrupt line (or whatever mechanism is connected) is
> shared
> with another device. My ISR (for a PCIe device) is constantly called in
> most
> systems (I’ve always assumed it was sharing with the network port,
> graphics
> driver or other device). I check (and clear) the Interrupt Status Register
> for
> my device and return TRUE if it was me (passing the Reason for Interrupt
> to the
> DPC). If the status isn’t set it wasn’t my interrupt, I return FALSE.
>
> If I stop and start the driver it seems I’m placed at the end of the queue
> (and
> so I don’t get called unless the interrupt wasn’t claimed by an ISR
> further up
> the queue).
>
> I have to admit I’ve always found this way of handling the interrupt-stack
> a bit
> weird. Surely it would be more efficient to call all the ISRs in the queue
> regardless of whether earlier drivers “claimed” the interrupt or not - two
> interrupts at once is entirely possible.
>
> –
> Graeme Griffiths
> Image Processing Techniques Ltd.
> +44 1256 345 927
>
> The contents of this Email and any files transmitted with it are
> confidential
> and intended solely for the use of the individual or entity to whom it is
> addressed.
> The views stated herein do not necessarily represent the view of the
> company.
> If you are not the intended recipient of this Email you may not copy,
> forward,
> disclose or otherwise use it or any part of it in any form whatsoever.
> If you have received this mail in error please Email the sender.
>
> Image Processing Techniques Ltd
> Tel : +44(0)1256345910
> Fax : +44(0)1256345911
> Website : www.imageproc.com
> Registered Office : Unit 3, Intec 2, Wade Road, Basingstoke, Hampshire,
> RG24 8NE
> Registration Number : 3564291
> Place of Registration : England
> VAT Registration : 709198411
>