Win10: Interrupt miss in application

Without seeing source code for SendDeviceIoctl, there’s little we can do. Nothing in the code you posted actually sends an ioctl or waits for a result.

You still haven’t described how you know interrupts are being “missed”. I assume you’re getting them all in kernel mode, so is it your user-mode accounting that comes up short?

this code does not make sense

while (1)
{
WaitForDwellEvent (INFINITE, &Counter);
//Check Counter
WaitForDwellEvent (0, &Temp); //Send IOCTL - Do not wait
}

looking at nothing else in your code, this does not follow any valid pattern.

for overlapped IO, there are two general patterns - top down or bottom up. The ‘top down’ pattern is the classic one where the application determines what to do and the kernel does it and tells you when its done. A typical example is writing to a file. The timing of when a certain activity should start is entirely under the control of the UM process.

the ‘bottom up’ pattern is where the what to do is still controlled by the UM application, but the when to do it is under the control of the KM component. either within itself, or in response to some hardware or network event.

with the top down pattern, IO is initiated from UM when work needs to be done. you do not expect any special queueing on the driver side because the driver’s job is to do the work ‘now’. This loop cannot implement that pattern.

with the bottom up pattern, IO is initiated from UM in advance of when the work is to be done by the driver. in this pattern the driver is expected to queue and pend multiple IOCTLs from UM, and then complete them ‘later’. This loop also does not implement this pattern, as further IOCTLs submitted from UM to KM should happen in the completion handler and not be in any kind of loop.

but in my shop, while(1) would be a gigantic red flag. While(true) and proper code formatting (a matter of opinion)

reading between the lines, I am almost sure that when you say that events are being ‘missed’, they happen during times where you driver does not have a pending IOCTL to complete because this loop cannot ever sent enough.

Hello Tim, MBond2, All,

The attached Gwbr.cpp.txt contains also the code for SendDeviceIoctl
The WaitForDwellEvent sends an IOCTL to the driver which is handled in Control.c.txt (line 119)
The IOCTL is forwarded to a queue allocated only for this IOCTL code and the request is completed in IsrDps.c.txt (line 210)

In IsrDpc.c.txt, upon getting a “dwell” interrupt, a request is pulled from this queue (line 200) and completed.

The tip I got in another thread: https://community.osr.com/discussion/292320/calling-deviceiocontrol-from-multiply-threads#latest
was to send few IOCTLs without waiting during initialization.
The IOCTL waits for ever till a “dwell” interrupt is received. Only then, one more IOCTL is sent without waiting .

The method for checking a missed interrupt in the application:
Upon “dwell” interrupt, a counter is incremented. This counter is sent back to the application upon an interrupt.
The reply is built in IsrDpc.c.txt (line 207)

Thank you,
Zvika

I guess we can infer that you have fixed your problem. we can also infer that your pattern is the ‘bottom up’ pattern and you have correctly identified that pending multiple IRPs allows your driver to do something useful when these interrupts happen during times when your previous solution did not have any pending IO.

next you need to consider how many pending IRPs you should have. This is a problem that you likely cannot exactly solve, but it depends on how important it is to ‘never’ miss an event and what happens when the system is under load and the UM components cannot process completions and send new IRPs for a ‘long’ time

An interrupt-driven driver always needs to be able to handle the case where a new interrupt arrives before the previous interrupt has finished processing, especially if interrupts arrive quickly. Are you handling that case? You can’t assume a nice linear progression from ISR to DPC to user-mode.

IOW, you are not guaranteed a 1:1 correspondence between calling WdfInterruptQueueDpcForIsr and calls to your DPC. Not to mention, your DPC can run in parallel on multiple CPUs simultaneously.

Peter

Hello Tim, MBond2, Peter, All,

Thank you very much for your replies.

Can you please offer a mechanism (or a sample code) to handle a case in which new interrupt arrives before previous interrupt finished processing ?
How should I handle a not 1:1 correspondence between calling WdfInterruptQueueDpcForIsr and calls to your DPC ?

Best regards,
Zvika

How should I handle a not 1:1 correspondence between calling WdfInterruptQueueDpcForIsr and calls to your DPC ?

I get this question almost every time I teach our WDF seminar. And I frankly never know how to answer it. The answer is: “You write some code. That is, you write whatever code you need to write to handle this situation, in your driver, based on your hardware. I can’t tell you what that is.”

This code might be something as simple as keeping a count of unserviced interrupts in your ISR, and decrementing that count to zero in your DPC.

Or, maybe that code entails you saving some state away (say, the latched state of your device’s registers) on each ISR invocation… and retrieving that saved state in your DPC and processing it. This save/retrieve might be something as simple as non-destructively OR’ing register status into copy of one or more of your hardware registers (we call these “shadow registers”) that you maintain in (for example) your Interrupt context. Or the save/retrieve might be something as complex as storing a bunch of stuff into a data area and then making an entry on a linked-list (with ExInterlockedInsertTailList) for each ISR invocation, and retrieving and processing all those entries (using ExInterlockedRemoveHeadList) once you get to your DPC.

Only you know what you need to do, based on your hardware and driver architecture. The key thing is you cannot assume that your calls to WdfInterruptQueueDpcForIsr and calls to your DpcForIsr will be 1:1… you have to understand that these can be N:1… and write whatever code you need to write to ensure that your driver “does the right thing” under this constraint.

Peter

The interrupt tells you “something interesting happened”. It should not be giving you any more detail than that. It is up to the driver to determine what happened. If some data finished transferring, then your driver needs to be able to find out how much data there was. If your driver has to assume that each interrupt means 64k bytes transferred, then your design is badly broken.

In addition to all of the coalescing that can happen in software, interrupts can be coalesced in hardware too. Message signaled interrupts have been the norm for many years now and the hardware will queue exactly 1 and discard any duplicates.

the point is the same: no matter how you get to your DPC, n things of interest have happened on your hardware, and you have n things to report back up to UM or wherever. and if you don’t have at least n pending IRPs to complete, then you have to decide what to do. Throw the data away or queue it up (to some limit). but as Tim points out, if you can’t figure out how to calculate n, or get the details of all n things, you are sure to have data loss before you even worry about how the UM program calls the driver.

certain conditions will inevitably result in data loss. so then it becomes an engineer’s job to determine acceptable thresholds. If you anticipate an interrupt frequency on the order of 1 Hz, than just about any design that works at all will work in nearly every case. But even then, there is still no reason to do it deliberately badly

Hello,

I think I have further information:

As I wrote, during init, I’m sending 10 IOCTL requests with the code: DWELL_INTERRUPT_REQUEST_CODE
I checked that those requests are handled.
For each of them, the message is sent to a queue.

Upon interrupt, a message is read from this queue in DPC using WdfIoQueueRetrieveNextRequest
In the application, after this IOCTL is answered, another IOCTL is sent.

In case of “interrupt miss”, WdfIoQueueRetrieveNextRequest returnes with 0x8000001A which means: STATUS_NO_MORE_ENTRIES

How can this happen ?
Is it possible that the queue is too small ?

Thank you,
Zvika

Your app hasn’t sent enough entires. Have the app send 100 instead of 10 entries. See if that works.

Peter

Hello,

I had a BUG in the application.
In SendDeviceIoctl (line 92) , if the timeout is 0, rc=WAIT_TIMEOUT and the event is not inserted back to the pool.
After fixing it, I’m not getting STATUS_NO_MORE_ENTRIES from WdfIoQueueRetrieveNextRequest in IsrDpc.c

The cause to the miss:
The time from one interrupt to another is 20msec.
But the time required to process this event in the application can take more than 20msec.

Thank you very much for the detailed answers.
I really appreciate your help.

Zvika

you do understand that you can certainly get this failure again even after you have fixed whatever bug you had. It is inherent in the nature of the system and you should have some kind of strategy to deal with it. many times, the strategy is simply ‘i need 4 buffers pended to handle normal load, so i’ll, pend 4,000 and if things go that far bad then i’ll just raise an error’. that is a valid strategy. It is not valid to say I need 4 and so i’ll pend 8 and cross my fingers

Hello MBond2,

Thank you very much !

Hello,

At the current design, application is using IOCTL to block till interrupt is received in kernel.
The IOCTL coming from application is immidiately sent to internal queue.
When the interrupt is handled in the DPC stage, the IOCTL request is pulled from this queue and answered.

I would like to suggest an alternative:
Upon interrupt, a small descriptor will be written to internal queue during DPC stage.
IOCTL handler will try to pull a descriptor from the internal queue.
In case the internal queue is empty, the pull will be blocked.

Can you please tell is this alternative can work better ?

Thank you,
Zvika

consider how you propose to block in your DPC in a way that will allow UM code a chance to run and send down a new IRP. If there are ‘enough’ cores, one of them might be available for the UM code to execute while you hog this one. But what if there aren’t? An what about the huge security hole where a misbehaving UM process can starve the system of resources and possibly lock it up completely

having a bounded length queue of data that has been received from hardware but did not have a ready IRP to complete is a valid strategy to deal with this kind of problem. But it has to have a maximum length and can’t block the DPC when it overflows. Many flow control algorithms exist to handle this problem, but the much better answer is to always have an IRP pending for completion

1 Like

the much better answer is to always have an IRP pending for completion
Or even mix of two. Depends on the OP"s actual h/w design and requirements, of course.
For example if the requirement is just not to miss interrupts (but each interrupt by itself does not carry any data) the driver can simply count the interrupts and pass the count.

Can you please tell is this alternative can work better ?

How is this functionally any different from just using the IRP, and pulling it off a manual queue in the DPC?

As Mr. MBond2 said… just keep enough IRPs on the manual Queue.

This isn’t complex, or unusual, Mr. @zvivered … this is how everyone does this. Just make your user-mode code work properly.

Peter

Hello MBond, Pavel and Peter,

Thank you very much !

Zvika