Possible to miss a MSI ?

NTDEV
1

Hello all,

Imagine a hardware that triggers an interrupt each time n bytes of data are available for reading.

With a level-triggered interrupt, it’s possible for this hardware to know that a previous interrupt was not cleared by the driver at the moment a new interrupt should occur. Then, the hardware can trigger a specific error interrupt, so that the driver knows that it missed one or several interrupts (then, the driver can, for example, read a specific register on the hardware to know how many bytes are available for reading).

But for an edge-triggered interrupt (or for a MSI, I guess it’s the same), how can the hardware or the driver know that an interrupt was lost/merged ?

Thanks in advance.
Best regards,
Vincent

2

Perhaps your hardware could have a register that would be incremented each time the hardware triggers an interrupt. The driver would read that register, so that it would know how many interrupts were triggered from the last time the driver ISR was executed. The hardware could clear the interrupt counter when it is readed by the driver.

Hopes this helps and excuse my English.

3

> With a level-triggered interrupt, it’s possible for this hardware to know that a previous interrupt

was not cleared by the driver at the moment a new interrupt should occur. Then, the hardware can trigger
> a specific error interrupt,

Well, as long as the line via which this piece of hardware signals interrupts remains asserted it will just have no chance to issue a new interrupt of any description, don’t you think…

But for an edge-triggered interrupt (or for a MSI, I guess it’s the same), how can the hardware
or the driver know that an interrupt was lost/merged ?

From the state of device registers, don’t you think…

Anton Bassov

4

First, thanks for your answers Eugenio and Anton.

Eugenio,
Anton,

From the state of device registers, don’t you think…
That’s what I wanted to avoid.
We lose one advantage of MSI against INTx : avoid reading device registers when processing an interrupt and save CPU clock cycles.
But I guess this is the only solution : and the read can be done inside DPC, leaving ISR free of PCI access.

Anton,

Well, as long as the line via which this piece of hardware signals interrupts remains asserted it will just have no chance to issue a new interrupt of any description, don’t you think…
In fact, when the driver will clear the source of the Data interrupt, then its ISR will be called again because the source of the Error interrupt is not yet cleared and the hardware still keeps the line asserted.

Vincent

5

Technically, a properly written driver will never “loose” an interrupt. The caveat there is that if interrupts really are occurring faster than the OS can handle them, then you need to be looking for another OS, or re-consider the interrupt design on your card. However, h aving said that, in most cases, “lost” interrupts occur because of poor design in the ISR/DPC interfaces. Your interrupt handler should be designed to process your hardware till all interrupts are honored, queue as much as possible to a DPC and only do only absolutely necessary processing in the ISR.

Gary G. Little

----- Original Message -----
From: “vincent saint-martin”
To: “Windows System Software Devs Interest List”
Sent: Wednesday, January 5, 2011 3:22:57 AM
Subject: [ntdev] Possible to miss a MSI ?

Hello all,

Imagine a hardware that triggers an interrupt each time n bytes of data are available for reading.

With a level-triggered interrupt, it’s possible for this hardware to know that a previous interrupt was not cleared by the driver at the moment a new interrupt should occur. Then, the hardware can trigger a specific error interrupt, so that the driver knows that it missed one or several interrupts (then, the driver can, for example, read a specific register on the hardware to know how many bytes are available for reading).

But for an edge-triggered interrupt (or for a MSI, I guess it’s the same), how can the hardware or the driver know that an interrupt was lost/merged ?

Thanks in advance.
Best regards,
Vincent

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

6

But when using MSIs, there is nothing to do inside the ISR, since the source of the interrupt is contained in the message ID (no need to read the device to know the source and clear it).
In this case, the only thing to do is queuing a DPC.
Am I wrong ?

7

Whether or not you are wrong depends on the hardware design.

If your hardware works by looking for work in message queues stored in a
common buffer, updating those work queues as it goes, then you have nothing
to do in your ISR. You can infer whatever you need from the state of those
datastructures in memory.

If, however, your hardware is where the state is, you probably need to touch
your hardware in your ISR, and probably your DPC. Message-signaled
interrupts only solve part of that problem.

Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…

But when using MSIs, there is nothing to do inside the ISR, since the source
of the interrupt is contained in the message ID (no need to read the device
to know the source and clear it).
In this case, the only thing to do is queuing a DPC.
Am I wrong ?

8

xxxxx@cea.fr wrote:

Imagine a hardware that triggers an interrupt each time n bytes of data are available for reading.

With a level-triggered interrupt, it’s possible for this hardware to know that a previous interrupt was not cleared by the driver at the moment a new interrupt should occur. Then, the hardware can trigger a specific error interrupt, so that the driver knows that it missed one or several interrupts (then, the driver can, for example, read a specific register on the hardware to know how many bytes are available for reading).

But for an edge-triggered interrupt (or for a MSI, I guess it’s the same), how can the hardware or the driver know that an interrupt was lost/merged ?

In general, the hardware should not worry about this. What could you
possibly do with that knowledge? The interrupt signals that something
is available. The driver reads whatever is available. If the driver
doesn’t get around in time, the device’s FIFOs fill. That’s when the
hardware knows something has gone awry.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

9

I’ve had this question come up several times. RTOS people want to be able
to set the device priority, and the last time I looked at this, Windows does
not really provide an effective solution to this problem. So the interrupts
may not be coming in faster than the OS can handle them, but are coming in
faster than the current interrupt configuration allows (e.g., imagine a card
with a 20us response window, but was the lowest priority on the bus, and
several other ISRs could be preempting it).

Is the new driver world able to support requests to set device priority, or
is this still fixed by the PCI BIOS?

Note that “bursty” devices that need fast response for brief periods and
then go idle for a few hundred microseconds work well with deferred
queueing, but devices requiring constant high-frequency response often don’t
work well in Windows.
joe

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gary G. Little
Sent: Wednesday, January 05, 2011 10:47 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Possible to miss a MSI ?

Technically, a properly written driver will never “loose” an interrupt. The
caveat there is that if interrupts really are occurring faster than the OS
can handle them, then you need to be looking for another OS, or re-consider
the interrupt design on your card. However, h aving said that, in most
cases, “lost” interrupts occur because of poor design in the ISR/DPC
interfaces. Your interrupt handler should be designed to process your
hardware till all interrupts are honored, queue as much as possible to a DPC
and only do only absolutely necessary processing in the ISR.

Gary G. Little

----- Original Message -----
From: “vincent saint-martin”
To: “Windows System Software Devs Interest List”
Sent: Wednesday, January 5, 2011 3:22:57 AM
Subject: [ntdev] Possible to miss a MSI ?

Hello all,

Imagine a hardware that triggers an interrupt each time n bytes of data are
available for reading.

With a level-triggered interrupt, it’s possible for this hardware to know
that a previous interrupt was not cleared by the driver at the moment a new
interrupt should occur. Then, the hardware can trigger a specific error
interrupt, so that the driver knows that it missed one or several interrupts
(then, the driver can, for example, read a specific register on the hardware
to know how many bytes are available for reading).

But for an edge-triggered interrupt (or for a MSI, I guess it’s the same),
how can the hardware or the driver know that an interrupt was lost/merged ?

Thanks in advance.
Best regards,
Vincent

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
–
This message has been scanned for viruses and
dangerous content by http:</http:> MailScanner, and is
believed to be clean.

10

> In fact, when the driver will clear the source of the Data interrupt, then its ISR will be called again

because the source of the Error interrupt is not yet cleared and the hardware still keeps the line asserted.

Oh, I see, you mean raising the same interrupt for a different reason and then checking the reason for interrupt in ISR…

In order to realize that this is not the right solution in your situation regardless of interrupt type all you have to do is to consider the scenario when you fail to retrieve data more than once before your first invocation - at this point your problem gets simply reformulated as “how many times error occurred”, which is equivalent to the original " how much data am I missing" presentation. In other words, the problem remains the same…

Anton Bassov

11

The “new driver world” (or, at least, drivers for NT-based OSes running on APIC-based machines since 1995) get their interrupt priority pretty randomly. The PCI BIOS has nothing to do with it.

Up through NT4, the first driver to call HalGetInterruptVector got a high priority. The next one was lower, etc., until it cycled back to high again. At the time, your IDE driver tended to have very low tolerance for interrupt latency and it tended to start first.

From Windows 2000 onward, it was a function of some pretty odd PnP manager side effects.

Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

“Joseph M. Newcomer” wrote in message news:xxxxx@ntdev…
I’ve had this question come up several times. RTOS people want to be able to set the device priority, and the last time I looked at this, Windows does not really provide an effective solution to this problem. So the interrupts may not be coming in faster than the OS can handle them, but are coming in faster than the current interrupt configuration allows (e.g., imagine a card with a 20us response window, but was the lowest priority on the bus, and several other ISRs could be preempting it).

Is the new driver world able to support requests to set device priority, or is this still fixed by the PCI BIOS?

Note that “bursty” devices that need fast response for brief periods and then go idle for a few hundred microseconds work well with deferred queueing, but devices requiring constant high-frequency response often don’t work well in Windows.
joe

--------------------------------------------------------------------------------
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Gary G. Little
Sent: Wednesday, January 05, 2011 10:47 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Possible to miss a MSI ?

Technically, a properly written driver will never “loose” an interrupt. The caveat there is that if interrupts really are occurring faster than the OS can handle them, then you need to be looking for another OS, or re-consider the interrupt design on your card. However, h aving said that, in most cases, “lost” interrupts occur because of poor design in the ISR/DPC interfaces. Your interrupt handler should be designed to process your hardware till all interrupts are honored, queue as much as possible to a DPC and only do only absolutely necessary processing in the ISR.

Gary G. Little

----- Original Message -----
From: “vincent saint-martin”
To: “Windows System Software Devs Interest List”
Sent: Wednesday, January 5, 2011 3:22:57 AM
Subject: [ntdev] Possible to miss a MSI ?

Hello all,

Imagine a hardware that triggers an interrupt each time n bytes of data are available for reading.

With a level-triggered interrupt, it’s possible for this hardware to know that a previous interrupt was not cleared by the driver at the moment a new interrupt should occur. Then, the hardware can trigger a specific error interrupt, so that the driver knows that it missed one or several interrupts (then, the driver can, for example, read a specific register on the hardware to know how many bytes are available for reading).

But for an edge-triggered interrupt (or for a MSI, I guess it’s the same), how can the hardware or the driver know that an interrupt was lost/merged ?

Thanks in advance.
Best regards,
Vincent

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
–
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

12

> The ???new driver world??? (or, at least, drivers for NT-based OSes running on APIC-based

machines since 1995) get their interrupt priority pretty randomly. The PCI BIOS has nothing to do with it.

Actually, by the “new driver world” in Joe’s post I understood MSI, which, indeed, theoretically allows you to avoid some limitations that asserting a line implies and that BIOS (even ACPI one) is not guaranteed to allow the OS to resolve. Consider two devices sharing a line because of being physically wired together…

Anton Bassov

13

A lot of thinks are unclear to me about MSI :frowning:
I’m not talking about MSI-X for the moment.
I read the MSI.doc document from Microsoft and a part of the PCI 3.0 specifications and it’s still unclear.
A PCIe device is supposed to support MSI and simulated legacy INTx.
Imagine I write a KMDF driver that supports MSI and fallback to INTx.
If the PCIe device is driven under Windows XP, the device will be told to use INTx and an ISR will be called upon interrupt.
If the PCIe device is driven under Windows 7, the device will be told to use MSI and an IMSR will be called upon interrupt.
Now, imagine the PCIe device requests 2 different messages ; Windows 7 can allocate 1 or 2 messages (depending on available resources).
If only 1 message is allocated, then the MessageID parameter of the IMSR will always be 0.
Right ?
In that case, I need to read at least one device register to know the source of the interrupt.
What is the advantage of MSI over INTx in that case ?

Thanks.
Vincent

14

You don’t have to share your interrupt. With INTx, there are usually only
four vectors that must be shared between all devices. Some chipsets have
eight. With MSI, with only one message, you still get your own vector.

In almost all practical situations, though, you’ll get all your messages.
Only when there are too many devices plugged into the machine will you need
to use just one message.

With that said, there are few reasons to design a device with multiple MSI
messages. All multi-message MSI gets you is first-level interrupt dispatch
in hardware instead of software. If your device has 23 possible internal
interrupt sources in four categories, you can ask for four messages and then
your ISR knows the category before it touches hardware. This is nice, but
it rarely makes a difference in performance in practice.

Where it does make a difference is when your device is actually several
logical units, each with its own driver. Then it’s really nice to split
into multiple messages so that the drivers don’t have to co-ordinate with
each other. Even then, though, it’s likely that the rest of the device has
enough commonality between the logical units that you have to co-ordinate to
some extent.

As a final note, almost all the devices that I see that don’t support MSI-X
are supporting MSI simply because they have a requirement to do so per the
spec. When this is true, their interrupt logic tends to be centered around
a level-triggered usage model. It’s usually not the case that
level-triggered semantics translate well to edge-triggered MSI, and commonly
the solution involves examining device hardware to clear interrupting
conditions in the ISR, which mostly negates the point of multiple messages.

In short, MSI is often a poor match for real-world conditions (particularly
given how Intel’s front-side busses work - see Volume 3B, Chapter 8) and so
the SIG had to go design MSI-X. MSI-X isn’t perfect either, but at least
it’s flexible enough to be more useful.

Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…

A lot of thinks are unclear to me about MSI :frowning:
I’m not talking about MSI-X for the moment.
I read the MSI.doc document from Microsoft and a part of the PCI 3.0
specifications and it’s still unclear.
A PCIe device is supposed to support MSI and simulated legacy INTx.
Imagine I write a KMDF driver that supports MSI and fallback to INTx.
If the PCIe device is driven under Windows XP, the device will be told to
use INTx and an ISR will be called upon interrupt.
If the PCIe device is driven under Windows 7, the device will be told to use
MSI and an IMSR will be called upon interrupt.
Now, imagine the PCIe device requests 2 different messages ; Windows 7 can
allocate 1 or 2 messages (depending on available resources).
If only 1 message is allocated, then the MessageID parameter of the IMSR
will always be 0.
Right ?
In that case, I need to read at least one device register to know the source
of the interrupt.
What is the advantage of MSI over INTx in that case ?

Thanks.
Vincent

15
16

xxxxx@osr.com wrote:


> With INTx, there are usually only four vectors that must be shared between all devices. Some chipsets have eight.
>

Hmmmm… are there specific chipsets or systems with which you can know in advance will have 8??

The PCI bus only has 4 wires for this, so it would require two buses and
some magic in the PCI controller. Not rocket science, but certainly
very specialized.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

17

Depends on what you mean by “advance.” Just look at the ACPI namespace.
You’ll see them listed as “link nodes.”

Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…

18

No, Tim, the PCI bus has zero wires for this. It’s not defined as part of
the bus spec.

Each individual PCI slot (assuming physical PCI, which we aren’t talking
about in this thread) then there are four wires in the slot.

Each individual PCI function has a choice between 4 wires (virtual or
physical).

But it’s up to the motherboard implementer whether the four wires to any
specific slot have anything to do with each other. In many (older) servers,
they don’t.

With PCI Express, the virtual INTx message goes up to the root port. The
root port turns it into some other kind of signal (physical or virtual) that
attaches to the interrupt subsystem. Many chipsets map all INTx messages
onto four external signals. This is arbitrary, and decided by the chipset
designer.

In a machine where the chipset designer decided to map the INTx messages
onto 20 external signals per root port and there were 20 root ports, there
would be 400 interrupts available to PCI, assuming you had enough I/O APICs
in the system to have 400 interrupts. (The last part of that was specific
to x86, x64 and Itanic, not ARM. ARM doesn’t use APICs. Good luck finding
an ARM with PCI.)

Jake Oshins
Hyper-V I/O Architect (former interrupt guy)
Windows Kernel Group

This post implies no warranties and confers no rights.

“Tim Roberts” wrote in message news:xxxxx@ntdev…

xxxxx@osr.com wrote:


> With INTx, there are usually only four vectors that must be shared between
> all devices. Some chipsets have eight.
>

Hmmmm… are there specific chipsets or systems with which you can know in
advance will have 8??

The PCI bus only has 4 wires for this, so it would require two buses and
some magic in the PCI controller. Not rocket science, but certainly
very specialized.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

19

> In a machine where the chipset designer decided to map the INTx
messages

onto 20 external signals per root port and there were 20 root ports,
there
would be 400 interrupts available to PCI, assuming you had enough I/O
APICs
in the system to have 400 interrupts. (The last part of that was
specific
to x86, x64 and Itanic, not ARM. ARM doesn’t use APICs. Good luck
finding
an ARM with PCI.)

Are you saying that ARM doesn’t use PCI? It might be unusual to find an
ARM board with a full sized PCI slot but there are plenty with mini-PCI
and mini-PCI-e. Is mini-PCI that different?

James

20

> With a level-triggered interrupt, it’s possible for this hardware to know that a previous interrupt was

not cleared by the driver at the moment a new interrupt should occur. Then, the hardware can trigger a
specific error interrupt

This seems to be against the common sense of level-triggered interrupts.

The OS guarantees that, if the INT line is still asserted, then the ISR will be called once and once again.

Each ISR invocation must just analyze the hardware state and decide what to do.

The hardware’s task is only to assert the INT line due to some causes (there can be several), and have both the global interrupt disable register and maybe local per-cause disable register. Nothing else.

In this picture there is no such thing as “missing an interrupt”. Regardless of anything, if the hardware has the cause to interrupt, it will persist till it will be handled, i.e. it will be enough to re-invoke the ISR again and again till it will handle it.

As about MSIs - can you miss a memory write? :slight_smile:

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com