Interrupt Question

Hi,

I am using the following code to work with interrupts:
NTSTATUS VkEvtDriverDeviceAdd(__in WDFDRIVER Driver, __in PWDFDEVICE_INIT DeviceInit)
{

WDF_INTERRUPT_CONFIG_INIT(&interruptConfig, VkEvtInterruptIsr, VkEvtInterruptDpc);

interruptConfig.EvtInterruptEnable = VkEvtInterruptEnable;
interruptConfig.EvtInterruptDisable = VkEvtInterruptDisable;
interruptConfig.AutomaticSerialization = TRUE;

status = WdfInterruptCreate(device,&interruptConfig,WDF_NO_OBJECT_ATTRIBUTES,&pDevExt->Interrupt);


}

BOOLEAN VkEvtInterruptIsr(__in WDFINTERRUPT Interrupt, __in ULONG MessageID)
{
PVK_DEVICE_EXTENSION pDevExt;

pDevExt = VkGetDeviceExtension(WdfInterruptGetDevice(Interrupt));

WdfInterruptQueueDpcForIsr(pDevExt->Interrupt);
}

void VkEvtInterruptDpc(__in WDFINTERRUPT Interrupt,__in WDFDEVICE Device)
{
NTSTATUS status;

//Acquire this device’s InterruptSpinLock.
WdfInterruptAcquireLock(Interrupt);

// Release our interrupt spinlock
WdfInterruptReleaseLock(Interrupt);

return;
}

Now I have the following questions:
Q1:
In the DeviceAdd callback, there is the line interruptConfig.AutomaticSerialization = TRUE;
In the InterruptDpc callback, there is the line WdfInterruptAcquireLock(…)
When AutomaticSerialization is set to TRUE, is it still necessary to acquire a lock?
I consulted the book Developing Drivers with WDF, but haven’t found a satisfying answer.

Q2:
In the InterruptISR callback, is it necessary to check if the interrupt is for our device?

Q3:
Is it necessary to disable the interrupt in the ISR routine and reenable in the DPC?

Q4:
Are there any caveats or do’s/dont’s when working with interrupts?

Thanks in advance,

Kurt

Whether you need to acquire the interrupt spin lock depends on your hardware and the design of your driver. However, note that setting AutomaticSerialization to TRUE in the WDF_INTERRUPT_CONFIG structure doesn’t serialize the ISR itself… rather it results in the DpcForIsr being serialized (with your other event processing callbacks). Thus, in many cases, you’ll still need to use WdfInterruptAcquireLock to ensure that your DPC and ISR are serialized (such as when you need to access a set of registers atomically in your DPC).

Yes, assuming you’re device is not using MSI (which would be unusual, almost all devices are traditional line-based interrupt devices).

It depends on your hardware and the design of your driver.

What’s absolutely necessary is that – in your interrupt service routine – you get your device to stop interrupting (by indicating that the device’s interrupt is acknowledged).

A rather classic design for devices that have one request outstanding at a time is to acknowledge the interrupt in your ISR, and disable interrupts there.

Next, in your DpcForIsr, you interrogate the device’s registers to determine the reason for the interrupt and retrieve any information that you may need from the device. You use this information to complete the outstanding request (as appropriate). Having thus satisfied the request and serviced the device, you then start the next request on your device (if one is waiting) including re-enabling interrupts on the device. Then you return from your DpcForIsr.

The classic caveats are

  • You can’t call WdfRequestComplete from your ISR – this is a common newbie mistake.
  • Limit the processing you do in your ISR to things that are truly time-critical – everything else should be done in your DpcForIsr (that’s what it’s for).
  • Unless absolutely, positively, necessary do not try to cancel requests that are in-progress on your device – if requests on your device will always complete within a short period of time, there’s no need to worry about in-progress request cancellation, which brings with it a TON of hideous complications and latent bugs.

Peter
OSR

xxxxx@barco.com wrote:

Now I have the following questions:
Q1:
In the DeviceAdd callback, there is the line interruptConfig.AutomaticSerialization = TRUE;
In the InterruptDpc callback, there is the line WdfInterruptAcquireLock(…)
When AutomaticSerialization is set to TRUE, is it still necessary to acquire a lock?

No. When AutomaticSerialization is TRUE, the framework will grab the
spinlock for you. Note that this means that your interrupt cannot be
handled while your DPC is running, and that’s undesirable in many
circumstances.

I’ve said it before and I’ll say it again. KMDF makes it very, very
easy for you to over-synchronize yourself. When I did my first KMDF
driver, I naïvely thought that “if a little synchronization is good,
then a lot must be better”. As a result, I spent a lot of time chasing
down some very tricky deadlocks. In the end, I turned off
AutomaticSerialization. You really have to think about what you are
locking out.

Q2:
In the InterruptISR callback, is it necessary to check if the interrupt is for our device?

Yes. 99% of the time, a PCI interrupt is shared with other devices.
The I/O subsystem calls each driver that connected to that IRQ until one
of them claims the interrupt. You must be able to determine with
certainty whether it was YOUR device that triggered the interrupt.
That’s what the return value from the ISR is used for.

Q3:
Is it necessary to disable the interrupt in the ISR routine and reenable in the DPC?

Not generally, but to a certain degree it depends on your hardware. The
interrupt cannot fire while your ISR runs. And, if you use
AutomaticSerialization, it can’t fire while the DPC runs, either.

You need to understand how your hardware behaves. If it fires one
interrupt per millisecond, then there is clearly no need to disable the
interrupts. Your ISR and DPC are only going to last a few microseconds
(RIGHT?). If your hardware fires one interrupt per microsecond, then
you have to decide what you need to do. If you need to handle all of
the interrupts, then your ISR needs to be able to queue up the fact that
an interrupt occurred, so your DPC can pop them off and handle them in a
bunch (and in that case, you do NOT want AutomaticSerialization). If
you don’t really need to handle off of the interrupts, then you might
want to disable interrupts in the ISR to give you a chance to catch up.
But, in that case, I would wonder about your hardware design…

Q4:
Are there any caveats or do’s/dont’s when working with interrupts?

Oh, lots of them. :wink: There used to be a white paper on the WHDC web
site that described guidelines for handling interrupts, but I don’t have
the link now. One key point is that the ISR needs to run as quickly as
possible. While your ISR is running, you own the CPU. Many interrupts
are disabled. If your ISR runs for a long time, you not only kill
performance, but you can actually cause the clock to lose time.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

The device is using single-channel DMA (busmaster, scatter/gather).
The dataflow is as follows: inputbuffer -> processing by device -> outputbuffer -> interrupt.
So basically, the device is reading data from a userbuffer, does some processing, stores the processed data in an outputbuffer (also via DMA); on completion, it generates an interrupt. The driver catches the interrupt, reads the appropriate registers and completes the request. This is an atomic operation, since the device can handle only one request at a time. So, there can be only one outstanding request. Of course, it is possible that there are multiple DMA transfers. The number of interrupts/sec will be in the order of 10ths of millisecs. There is only one IO queue which is device synchronized.
In this case, what is then the approach to take? autoserialization=TRUE, interrupt disable (ISR), acquire lock, read registers, enable interrupt (DPC)?

The spin lock it’ll grab is the DISPATCH_LEVEL spin lock for serialing the DPC, not the ISR spin lock.

As I said in my reply, AutomaticSerialization = TRUE – when applied to the interrupt object – does NOT serialize execution of the DPC against the ISR. Rather, the affect of this is to serialize execution of the DpcForIsr associated with the ISR against the event processing callbacks.

Think about it… if AutoSerial = TRUE serialized the execution of the ISR with anything else, all those other things would have to run at DIRQL… and then you’ve have just about a whole driver (timer callbacks, DPC, etc) that ran at DIRQL. Not a good thing by any measure.

This is a VERY confusing and rather poorly documented feature of KMDF. I’ve had to go look it up numerous times (because I keep forgetting how this AutoSerial thing works).

Of course, there IS a chance that I misunderstand this… But I’m pretty sure that’s the way it works.

The whole AutoSerial=TRUE thing has to do with how you serialize between your DPC and other routines. This would be important for thing like managing your queue of pending requests, for example.

If you disable interrupts on your device in your ISR, there’s most likely no reason to acquire the interrupt spin lock in your DPC… because the goal of doing this would be to keep your DPC and ISR from running in parallel, and if interrupts are disabled on your device, your ISR isn’t going to run in any case (except if it’s called for ANOTHER device that you’re sharing the interrupt with, of course, but that shouldn’t matter to your DPC).

SO:

ISR:

  1. Check if your device is interrupting. If it is…
  2. Disable interrupts on your device
  3. Acknowledge the current interrupt on your device to allow the present interrupt to be dismissed
  4. Request your DPC

DpcForISR:

  1. Interrogate your device’s registers (or some other stored state) to determine why you’re in your DPC
  2. Service the device – by continuing or completing the current request
  3. Re-enable interrupts on your device
  4. Leave your DPC

Peter
OSR

Thank you Peter, it’s much clearer now.
One more question: between the time that WdfInterruptQueueDpcForIsr is called and the actual call of the DPC, is it possible that another IO request (IOCTL) gets serviced?
Or will the DPC be the first callback after the interrupt has been acknowledged (or does this depend on the autoserialization flag?)?
I definitely do not want anything to happen between the ISR and the DPC.

Yes, indeed. Absolutely. There’s reallly no way to prevent this.

Are you worried about your EvtIoDeviceControl event processing callback starting that next request before the first is complete? If so, this is a common programming pattern: Use a flag to indicate to your I/O event processing callbacks need to be aware that the current request is not yet complete, and therefore it should QUEUE any requests it receives, as opposed to INITIATING them.

Then, when you DpcForIsr has serviced the current request to completion, the DpcForIsr checks the queue of pending things to do, and if there’s something waiting, initiates the request on the device.

Peter
OSR

xxxxx@barco.com wrote:

The device is using single-channel DMA (busmaster, scatter/gather).
The dataflow is as follows: inputbuffer -> processing by device -> outputbuffer -> interrupt.
So basically, the device is reading data from a userbuffer, does some processing, stores the processed data in an outputbuffer (also via DMA); on completion, it generates an interrupt. The driver catches the interrupt, reads the appropriate registers and completes the request. This is an atomic operation, since the device can handle only one request at a time. So, there can be only one outstanding request. Of course, it is possible that there are multiple DMA transfers. The number of interrupts/sec will be in the order of 10ths of millisecs. There is only one IO queue which is device synchronized.
In this case, what is then the approach to take? autoserialization=TRUE, interrupt disable (ISR), acquire lock, read registers, enable interrupt (DPC)?

If there can only be one outstanding request at a time, then you don’t
really need AutomaticSerialization. You can’t send another request
until the previous request is completed, so there’s no opportunity for
overlap.

So, the ISR stops the device from interrupting and fires the DPC. The
DPC does whatever it needs to do to “complete” the request, then pulls
the next request from some queue, prepares it, and submits it to the device.

Right?


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

xxxxx@barco.com wrote:

Thank you Peter, it’s much clearer now.
One more question: between the time that WdfInterruptQueueDpcForIsr is called and the actual call of the DPC, is it possible that another IO request (IOCTL) gets serviced?
Or will the DPC be the first callback after the interrupt has been acknowledged (or does this depend on the autoserialization flag?)?
I definitely do not want anything to happen between the ISR and the DPC.

Why? In fact, the DPC will be higher priority, so it should run
immediately, but it’s not that hard to program around this. Your ioctls
will be putting tasks on a queue, and your DPC will be pulling tasks off
a queue. The interlocked doubly-linked list function or the WDF
collection functions are easy enough to use, and by using them it
shouldn’t MATTER in what order things happen.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

If you just have one item on the hardware at once, create a sequential
queue that feeds the requests into your dma logic. This way, not matter
how many requests are sent to the driver, only one is active at once.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Friday, July 27, 2007 3:48 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Interrupt Question

xxxxx@barco.com wrote:

Thank you Peter, it’s much clearer now.
One more question: between the time that WdfInterruptQueueDpcForIsr is
called and the actual call of the DPC, is it possible that another IO
request (IOCTL) gets serviced?
Or will the DPC be the first callback after the interrupt has been
acknowledged (or does this depend on the autoserialization flag?)?
I definitely do not want anything to happen between the ISR and the
DPC.

Why? In fact, the DPC will be higher priority, so it should run
immediately, but it’s not that hard to program around this. Your ioctls
will be putting tasks on a queue, and your DPC will be pulling tasks off
a queue. The interlocked doubly-linked list function or the WDF
collection functions are easy enough to use, and by using them it
shouldn’t MATTER in what order things happen.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

> Q2:

In the InterruptISR callback, is it necessary to check if the interrupt is
for our
device?

Yes.

Is it necessary to disable the interrupt in the ISR routine and reenable in
the >
DPC?

This is one of the possible ways of dealing with “ISR interrupting its DPC”
issue. There are another ones.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Doron, I have created ONE I/O queue that is device synchronized. Is this sufficient, or do you mean I have to create a second queue for the DPC requests?
Put in another way: if I have one and only one device synchronized queue and the current request involves a DMA transfer, will that request stay the current one until I complete the request (in the DPC probably)?
So, is the flow like: requestX … start DMA … (after some time) interrupt … dpc… complete request… requestY… aso
or can the flow be: requestX … start DMA … requestY… (after some time) interrupt (for requestX)… (before DPC for requestX) requestZ… dpc …?

Device synchronization means that only one callback is called at once,
but it does not mean that only one request is active at once. What type
(parallel, sequential) of queue did you create? Instead of relying
device synchronization consider using a sequential queue. Since the
queue controls the active request, there is no need to synchronize the
DPC against the queue EvtIo callback.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@barco.com
Sent: Saturday, July 28, 2007 12:49 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Interrupt Question

Doron, I have created ONE I/O queue that is device synchronized. Is this
sufficient, or do you mean I have to create a second queue for the DPC
requests?
Put in another way: if I have one and only one device synchronized queue
and the current request involves a DMA transfer, will that request stay
the current one until I complete the request (in the DPC probably)?
So, is the flow like: requestX … start DMA … (after some time)
interrupt … dpc… complete request… requestY… aso
or can the flow be: requestX … start DMA … requestY… (after some
time) interrupt (for requestX)… (before DPC for requestX) requestZ…
dpc …?


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Doron,

this is the code to setup the queue:
WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&attributes, VK_DEVICE_EXTENSION);

attributes.SynchronizationScope = WdfSynchronizationScopeDevice;
WDF_IO_QUEUE_CONFIG_INIT_DEFAULT_QUEUE(&ioQueueConfig, WdfIoQueueDispatchSequential);
ioQueueConfig.EvtIoDeviceControl = VkEvtIoDeviceControl;
status = WdfIoQueueCreate(pDevExt->Device,&ioQueueConfig,WDF_NO_OBJECT_ATTRIBUTES,&pDevExt->Queue);

As you see, there is one and only one sequential queue.
So, in this case I don’t need autoserialization=TRUE and there will be only one active request at a time?

Kurt

Correct, a sequential queue will guarantee that only one request is
active at once in the particular queue. If you only feed request from
this queue to your dma, then only one request will be active in your dma
engine.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@barco.com
Sent: Monday, July 30, 2007 12:26 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Interrupt Question

Doron,

this is the code to setup the queue:
WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&attributes,
VK_DEVICE_EXTENSION);

attributes.SynchronizationScope = WdfSynchronizationScopeDevice;
WDF_IO_QUEUE_CONFIG_INIT_DEFAULT_QUEUE(&ioQueueConfig,
WdfIoQueueDispatchSequential);
ioQueueConfig.EvtIoDeviceControl = VkEvtIoDeviceControl;
status =
WdfIoQueueCreate(pDevExt->Device,&ioQueueConfig,WDF_NO_OBJECT_ATTRIBUTES
,&pDevExt->Queue);

As you see, there is one and only one sequential queue.
So, in this case I don’t need autoserialization=TRUE and there will be
only one active request at a time?

Kurt


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer