Loosing Interrupts on a PCI Express card running under XP

NTDEV
21

> Hi All,

Some additional information here. I have gotten the failure to happen
on
a development machine running a debug version of the driver. I added
counters to the ISR and DPC for each reason that the ISR or DPC could
be
running, in addition to bringing out the InterruptReason array from
the
devExt.

The data at the time of failure looks as follows (these are just
counters):
Error ISR: 3
Dma ISR: 9998
Can ISR: 304900
Error DPC: 3
Dma DPC: 9998
Can DPC: 304900

The InterruptReason is all zeros, no interrupts pending from the ISR
for
the DPC.

(The 3 error ints and Dpcs are probably red herrings, this is just
saying
that I set up the test generator DMA incorrectly.)

Could you try also adding counters for:
ISR called total count
DPC called total count
ISR called but no reason found for ISR to be called
DPC called but no reason found for DPC to be called

If the first two counts are identical then you know for sure that you
have a 1 for 1 ratio of ISR and DPC calls, which will rule out some of
the corner cases that have been discussed here.

If the second two are != 0 it might tell you something…

Also, it may be diagnostically useful to create a timer which you
schedule from the DPC to run in (say) 500ms time which reports on what
it finds. The idea being that if things get ‘stuck’ then the timer will
report on the contents of your various registers and you could collect
data on the failure. It could even sync with the interrupt and process
the ‘missed’ interrupt if it thinks there is one, allowing the system to
continue and allowing you to collect more data. The only thing to watch
would be the case where there was 500ms of real inactivity and your
timer fired just before the ISR - it would see what looked like a missed
interrupt. You’d need to schedule a second timer or something to rule
that out…

James

22

> ISR called total count

DPC called total count
ISR called but no reason found for ISR to be called
DPC called but no reason found for DPC to be called

If ISR got called for no reason it means that interrupt has nothing to do with his device, which is absolutely normal scenario if iRQ is shared. Therefore, adding such counter will add to the confusion, instead of helping to detect a bug. When it comes to DPCs, the story is very different - indeed, matching interrupts
against DPC that can be either no-op, or,instead, “multireason”, or just “normal” ones may be of help here.
It is understandable that totals will match anyway, but their order may shed some light…

Anton Bassov

23

>

> ISR called total count
> DPC called total count
> ISR called but no reason found for ISR to be called
> DPC called but no reason found for DPC to be called

If ISR got called for no reason it means that interrupt has nothing to
do
with his device, which is absolutely normal scenario if iRQ is shared.
Therefore, adding such counter will add to the confusion, instead of
helping to detect a bug.

And if it’s not shared? I would assume (maybe incorrectly) that
debugging a device on a shared interrupt when there is a known interrupt
delivery problem is probably not the smartest thing to be doing. Sure,
test in a shared interrupt scenario when you know everything else is
working, but not when there is a known problem. My assumption therefore
is that in this case the interrupt is not shared (please clarify OP)

And if it’s not a shared interrupt then I think that an unexplained
‘nothing to do’ interrupt would be worthy of investigation.

When it comes to DPCs, the story is very
different - indeed, matching interrupts
against DPC that can be either no-op, or,instead, “multireason”, or
just
“normal” ones may be of help here.
It is understandable that totals will match anyway, but their order
may
shed some light…

The OP stated in his first email that it’s an “in-house designed PCI
Express card”. How willing are you to rule out a bug in the device
hardware? I’m imagining something along the lines of something going
wrong with interrupt delivery if control writes happen in a certain
order and it coincides with some internal state change.

I’m most likely wrong and there is just some obscure race in the code
somewhere, but it’s really frustrating spending time trying to fix what
you think is a software problem only to find out that the hardware is
misbehaving, so I think it’s worth adding a few extra counters just to
rule that out. And if you think hardware bugs are uncommon then have a
look through the Linux kernel some time (where all the driver source is
free to download) :slight_smile:

James

24

> I would assume (maybe incorrectly) that debugging a device on a shared interrupt when there

is a known interrupt delivery problem is probably not the smartest thing to be doing.

Do you really think you can always do something about that??? Don’t forget that there are just 4 PCI routing groups, but the number of PCI devices on a typical machine is well above that. Therefore,
in many cases you are just bound by the board wiring …

And if it’s not a shared interrupt then I think that an unexplained ‘nothing to do’ interrupt would be
worthy of investigation.

This is for sure…

Anton Bassov

25

>

> I would assume (maybe incorrectly) that debugging a device on a
shared
> interrupt when there
> is a known interrupt delivery problem is probably not the smartest
thing
> to be doing.

Do you really think you can always do something about that??? Don’t
forget
that there are just 4 PCI routing groups, but the number of PCI
devices on
a typical machine is well above that. Therefore,
in many cases you are just bound by the board wiring …

Hmmm… hadn’t thought of that. If it were me I would try and find a
board that had at least one slot that didn’t share an interrupt, but as
you say maybe that’s not so easy on a modern board.

James

26

> Hmmm… hadn’t thought of that. If it were me I would try and find a board that had at least one

slot that didn’t share an interrupt, but as you say maybe that’s not so easy on a modern board.

Just to give you an idea, one of the most common on-board USB configurations these days is EHCI with 4 companion UHCI controllers. It is understandable that two instances of UHCI cannot share a pin - otherwise ISR will have no chance to tell one instance from another. Therefore, all 4 routing groups are occupied - all other PCI devices have to share pin with an instance of UHCI, no matter if they are on board or in the slot…

Anton Bassov

27

So while all of this is true, you can alleviate this issue a little bit by disabling all devices in device manager who happen to share an INT. Now if one of those device is the boot HBA or something you need running, this does you no good

d

Sent from my phone with no t9, all spilling mistakes are not intentional.

-----Original Message-----
From: xxxxx@hotmail.com
Sent: Monday, February 23, 2009 3:13 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Loosing Interrupts on a PCI Express card running under XP

> Hmmm… hadn’t thought of that. If it were me I would try and find a board that had at least one
> slot that didn’t share an interrupt, but as you say maybe that’s not so easy on a modern board.

Just to give you an idea, one of the most common on-board USB configurations these days is EHCI with 4 companion UHCI controllers. It is understandable that two instances of UHCI cannot share a pin - otherwise ISR will have no chance to tell one instance from another. Therefore, all 4 routing groups are occupied - all other PCI devices have to share pin with an instance of UHCI, no matter if they are on board or in the slot…

Anton Bassov

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

28

I know you expressed doubt in a later message, but I wanted to follow this
up with a definitive statement for people searching the archives.

PCI Express devices in legacy interrupt mode do send messages on the bus for
interrupts. But the message protocol they use includes an acknowledge
message which tells the device to resample the interrupt and send a new INT#
message if the internal conditions for an interrupt are still met. This
makes it “level triggered,” semantically.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Team

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
>


>
> FIRST, thanks for taking the time to put together a well structure,
> informative, and clear post to the list. It’s a pleasure to try to answer
> such questions.
>
> BINGO! You’re right about PCI, but you’re on the PCI Express Bus.
>
> Even in legacy INTx mode PCI Express interrupts are generated via messages
> on the bus… so I believe (I’m not 100% sure, somebody will certainly
> correct me if I’m wrong… I’m not an expert on PCIe) you need to treat
> these also as if they had edge-triggered semantics.
>
> Can you check the Interrupt Status Register bits from your DpcForIsr, for
> example, and then service the various events that might be outstanding
> there?
>
> Your symptoms TOTALLY sound like you’re dropping an edge-triggered
> interrupt…
>
> Peter
> OSR
>
>
>

29

You’ve already gotten good advice from others, to use separate DPCs for
separate interrupt conditions. I’ll offer an alternative suggestion:

In your ISR, clear all the enable bits in your internal control register.
Now no more interrupts can occur from your device. In your DPC, loop
handling interrupt conditions until all the interrupt status bits are clear.
Re-enable the control bits on the way out of the DPC. You’ll get another
interrupt then, if one is pending.

If neither of these strategies works, it’s time to get out the bus analyzer
and see what’s actually happening.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Team

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
> Let me try and answer all the questions posed so far.
>
> First off, I really like the idea of queuing a DPC for each interrupt
> reason. I think this could simplify things a lot. And once I looked at
> the code for IoRequestDpc(), it seems easy enough to do. But before I do
> this, if it?s possible I?d like to understand where the logic error is
> first, if there is one, so I don?t make the same mistake again.
>
> I am working on a test version of the driver where I am going to pull out
> more of the interrupt related fields of the devExt structure, along with a
> few more statistics, since I can?t attach a debugger to some of the
> machines this occurs on.
>
> The logic behind the DPC was to make a local copy of the InterruptReason
> structure array while holding the ISR spinlock, zero out the devExt copy,
> and then drive all the DPC Logic from the local copy. If the ISR is
> called twice before the DPC is dequeued, the DPC should see both structure
> elements as Active and handle them in one pass. If a DMA interrupt
> occurs, the DPC starts to handle it, and then a CANbus interrupt occurs,
> the DPC should run twice in a row, or even at the same time on a
> multiprocessor system. If the DPC was queued and it had nothing to do,
> all 3 if statements should evaluate to false, and it should just fall
> through. The ?do more processing? at the end of the DPC is actually some
> clean-up code if one of the ?if? statements executed and had errors, so
> nothing should run there either in this case. At least, this is what I
> was going for. But it wouldn?t be the first time I?ve messed something
> up.
>
> The DisableXxxInterrupt and EnableXxxInterrupt functions are indeed
> wrappers to flip control register bits. The HW engineers used an
> interesting technique to guarantee bit setting and clearing is atomic, to
> set a bit, the high order bit must also be set. To clear a bit, the high
> order bit must be cleared. Bits that are 0 do nothing when the high order
> bit is set. Bits that are 1 do nothing when the high order bit is
> cleared. I think as long as writes to the PCI bus are not interleaved,
> I?m ok (I hope).
>
> The copies of the Control register are a leftover from when the HW
> engineer wasn?t sure if bits in there could change by themselves.
> (Something else to cut out next release I think).
>
> The logic to re-enable DMA was indeed cut out, because sometimes I don?t
> re-enable it if the transfer is complete, but yes, it lives inside the DMA
> if statement.
>
> Here?s the code for the Disable / Enable calls:
>
> VOID
> DisableCanbusInterrupts(PGEMINI_DEVICE_EXT DevExt)
> {
> ClearControlRegisterBits(DevExt, CTRL_BIT_CAN_INTERRUPT);
> }
>
>
> VOID
> DisableDmaInterrupts(PGEMINI_DEVICE_EXT DevExt)
> {
> ClearControlRegisterBits(DevExt, CTRL_BIT_DMA_INTERRUPT);
> }
>
>
> #define CTRL_MASK_SET 0x80000000
> #define CTRL_MASK_CLEAR (~CTRL_MASK_SET)
>
> VOID
> ClearControlRegisterBits( PGEMINI_DEVICE_EXT DevExt,
> ULONG Data)
> {
> //-----------------
> ULONG reg;
> //-----------------
>
> reg = Data & CTRL_MASK_CLEAR;
>
> WriteRegisterULongBar0(DevExt, OFFSET_CONTROL_REG, reg);
> return;
> }
>
>
> VOID
> WriteRegisterULongBar0( PGEMINI_DEVICE_EXT DevExt,
> ULONG Register,
> ULONG Data)
> {
> WRITE_REGISTER_ULONG((PULONG)((ULONG_PTR)DevExt->BAR0 + Register),
> Data);
>
> return;
> }
>
>
>

30

> In your ISR, clear all the enable bits in your internal control register. Now no more

interrupts can occur from your device.

Consider the following scenario.Device interrupts because of the hardware event X, and then, after handler stub enters execution but before device registers are checked in ISR, hardware event Y occurs. In this situation ISR will discover 2 hardware events after checking device registers, so that it will indicate this fact in the shadow registers. Therefore, even if you disable all lines in ISR, DPC still may discover that it got invoked for more than one reason, although disabling all lines in ISR, indeed, eliminates the possibility of no-op DPC.

I think the OP should start with breaking his DPC for ISR into 3 custom DPCs so that his DPCpart will work the same way regardless of his actions in ISR…

Anton Bassov

31

> you can alleviate this issue a little bit by disabling all devices in device manager who happen

to share an INT.

Good point , Doron- it just did not occur to me to think that even if your BIOS does not allow to separate other devices from interrupt line that your device uses , you still can simply disable them in Device Manager, effectively getting a dedicated IRQ…

Anton Bassov

32

Hi All,

Thanks for all the advice. I have some great suggestions for future improvements here!

I wanted to follow-up on what we discovered was happening, and what the root cause was, just for future list reference. As I mentioned, we have a driver that is talking to an in-house developed PCI Express card. Most of the functionality of the card lives inside a really big FPGA core, which uses a COTS PCI Express IP core to handle the bus interface.

(Note, when I say “firmware”, I am speaking of VHDL code inside the PCI Express card itself.)

The firmware interface to the interrupt mechanism of the IP core is via two logic lines, INT and INT_ACK. INT tells the core whether or not to assert an interrupt. INT_ACK was documented as a one clock acknowledge pulse which tells you whether or not it’s safe to change the logic on the INT line.

In later versions of the core’s documentation, the description changed to say that INT_ACK reflects the interrupt sense of the card on the PCI Express Bus. See the hole? When the interrupt is cleared by the firmware, it then incorrectly sees the state of INT_ACK as asserted and immedately flips to the state that allows the INT line to change again, but too soon. If another external event occurs within a few clock cycles of deasserting the INT line, the firmware sets it high again, thinking that it’s ok to do so. In actuality, this doesn’t allow the IP core enough time to send the Deassert Interrupt PCI Express bus message. Apparently, quickly toggling the INT line confuses the core (because we are no longer in spec with the hold time requirements), and the PCI Express Interrupt message is never sent, thus the missing interrupt.

The fix was a new version of firmware reflecting the correct logic from the newer updated documentation. (No driver changes needed, just VHDL code on the card.)

One other item, I mentioned in my first post that sometimes this problem just fixes itself. Here’s how that’s happening. We get locked up in firmware and don’t send an interrupt, but all our status bits indicate that we are / should be interrupting. Now, we are running with INTx semantics, so we could be sharing an interrupt line. If we are, and we are ahead of another device that is generating interrupts, when that device interrupts Windows calls us first, we see that we should be interrupting and claim the interrupt, and that gets us moving again.

As a correlary, when we install the driver after the system has booted, this problem becomes much more pronounced, because I’d guess that we are inserted at the tail end of the interrupt chain, so there isn’t anyone behind us to help us recover.

Thanks everyone for you help!

Best Regards,
-Mike

33

Thanks, Mike, for posting a follow-up to “close out” this thread. It’s much appreciated.

Peter
OSR