DPC queue draining

NTDEV
1

Hi all,

I am working on a driver that queues 1 or more DPC on each processor at
each time. As the driver (NDIS) needs to ensure timeliness, the normal
Importance level of DPC will not work as the Queue draining interrupt is not
called in-time causing a huge delay.

Thus I have to use KeSetImportanceDpc to set the importance of the DPC to
HighImportance, but this causes the DPC to be inserted not-in-order. I know
there is a MediumHighImportance but it is only available in Vista. Is there
anyway I can emulate that?

Alex

2

To be honest, I don’t think setting DPC importance may help you here…

The only reason why execution of a given DPC may get delayed is because the system is too busy processing interrupts and DPCs. Ironically, NIC seems to be the only device that is capable of delaying execution of DPCs due to high frequency of interrupts that it may generate.

If you disassemble HalEndSystemInterrupt() you will see that its actions depend on IRQL of interrupted code. If it was below DISPATCH_LEVEL, HalEndSystemInterrupt() calls KiDispatchInterrupt(), which handles DISPATCH_LEVEL software interrupts, i.e. is responsible for DPC queue draining and context switches. If IRQL was at or above DISPATCH_LEVEL, HalEndSystemInterrupt() simply returns, which means that DPC queue will get drained at some later stage, i.e. when IRQL drops below DISPATCH_LEVEL so that DISPATCH_LEVEL interrupt may fire.

Apparently, what happens in your case is that heavy network traffic makes your NIC interrupt frequently, and, hence, results in frequent DPC requests. Therefore, your NIC interrupts while DPCs that your ISR queued earlier are still getting processed, so that execution returns to interrupted code without a call to KiDispatchInterrupt(). In other words, a queue of DPC requests that have been initiated by NIC’s ISR gets longer and longer. Setting DPC priority does not seem to make sense under these circumstances, simply because all DPCs in the queue have been requested by your NIC’s ISR anyway, so that it just does not matter if the request goes to the head or tail of the queue. By speeding up processing of request X you delay processing of the request Y, which does not seem to make sense if these requests have been originated by the same device.

Anton Bassov

3

Hi Anton,

I see what you are getting at, but the thing is I am queuing DPCs with
different priorities (A timer routine DPC and a processing DPC where the
former has to ensure timeliness, thus its need to “speed-up” by preemption.

But that’s not the problem. The problem I am facing now is that it seems
that when I queue the timer routine DPCs every 1 ms on each processor, only
one processor will have DPCs fired every 1 ms interval. The other CPUs’
intervals will be around 10 ms or so. All this happens when the load is low.
It is only when I set the importance to HighImportance that the intervals
are corrected on every CPUs. This is undesirable as I do not want the DPC
being queued to preempt other HighImportance DPCs.

I hope I am making sense here.

Alex

On 6/27/07, xxxxx@hotmail.com wrote:
>
> To be honest, I don’t think setting DPC importance may help you here…
>
> The only reason why execution of a given DPC may get delayed is because
> the system is too busy processing interrupts and DPCs. Ironically, NIC seems
> to be the only device that is capable of delaying execution of DPCs due to
> high frequency of interrupts that it may generate.
>
>
> If you disassemble HalEndSystemInterrupt() you will see that its actions
> depend on IRQL of interrupted code. If it was below DISPATCH_LEVEL,
> HalEndSystemInterrupt() calls KiDispatchInterrupt(), which handles
> DISPATCH_LEVEL software interrupts, i.e. is responsible for DPC queue
> draining and context switches. If IRQL was at or above DISPATCH_LEVEL,
> HalEndSystemInterrupt() simply returns, which means that DPC queue will get
> drained at some later stage, i.e. when IRQL drops below DISPATCH_LEVEL so
> that DISPATCH_LEVEL interrupt may fire.
>
> Apparently, what happens in your case is that heavy network traffic makes
> your NIC interrupt frequently, and, hence, results in frequent DPC
> requests. Therefore, your NIC interrupts while DPCs that your ISR queued
> earlier are still getting processed, so that execution returns to
> interrupted code without a call to KiDispatchInterrupt(). In other words, a
> queue of DPC requests that have been initiated by NIC’s ISR gets longer and
> longer. Setting DPC priority does not seem to make sense under these
> circumstances, simply because all DPCs in the queue have been requested by
> your NIC’s ISR anyway, so that it just does not matter if the request goes
> to the head or tail of the queue. By speeding up processing of request X you
> delay processing of the request Y, which does not seem to make sense if
> these requests have been originated by the same device.
>
> Anton Bassov
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

4

Let me rephrase my previous mail.

When DPC importance is Medium, and DPC is targetted at current CPU or
non-targetted, the Queue-draining interrupt is always called, but when there
are multiple CPUs, and the DPC is targetted at different CPU, when DPC
importance is medium, the interrupt is called only when DPC queue length
exceeds maximum DPC queue length or when system is idle.

As discussed, it is undesirable to SetImportance to HighImportance as tasks
may keep getting preempted. And as I see it, there may be a few options
available:

  1. Set in registry the maximum DPC queue length for multiple CPUs (where
    to find the key?)
  2. Any call to dispatch the queue-draining interrupt manually.

I am new to kernel drivers, so please pardon me.

Alex

On 6/28/07, Alex Lee wrote:
>
> Hi Anton,
>
> I see what you are getting at, but the thing is I am queuing DPCs with
> different priorities (A timer routine DPC and a processing DPC where the
> former has to ensure timeliness, thus its need to “speed-up” by preemption.
>
> But that’s not the problem. The problem I am facing now is that it
> seems that when I queue the timer routine DPCs every 1 ms on each processor,
> only one processor will have DPCs fired every 1 ms interval. The other CPUs’
> intervals will be around 10 ms or so. All this happens when the load is low.
> It is only when I set the importance to HighImportance that the intervals
> are corrected on every CPUs. This is undesirable as I do not want the DPC
> being queued to preempt other HighImportance DPCs.
>
> I hope I am making sense here.
>
> Alex
>
>
> On 6/27/07, xxxxx@hotmail.com wrote:
> >
> > To be honest, I don’t think setting DPC importance may help you
> > here…
> >
> > The only reason why execution of a given DPC may get delayed is because
> > the system is too busy processing interrupts and DPCs. Ironically, NIC seems
> > to be the only device that is capable of delaying execution of DPCs due to
> > high frequency of interrupts that it may generate.
> >
> >
> > If you disassemble HalEndSystemInterrupt() you will see that its actions
> > depend on IRQL of interrupted code. If it was below DISPATCH_LEVEL,
> > HalEndSystemInterrupt() calls KiDispatchInterrupt(), which handles
> > DISPATCH_LEVEL software interrupts, i.e. is responsible for DPC queue
> > draining and context switches. If IRQL was at or above DISPATCH_LEVEL,
> > HalEndSystemInterrupt() simply returns, which means that DPC queue will get
> > drained at some later stage, i.e. when IRQL drops below DISPATCH_LEVEL
> > so that DISPATCH_LEVEL interrupt may fire.
> >
> > Apparently, what happens in your case is that heavy network traffic
> > makes your NIC interrupt frequently, and, hence, results in frequent DPC
> > requests. Therefore, your NIC interrupts while DPCs that your ISR queued
> > earlier are still getting processed, so that execution returns to
> > interrupted code without a call to KiDispatchInterrupt(). In other words, a
> > queue of DPC requests that have been initiated by NIC’s ISR gets longer and
> > longer. Setting DPC priority does not seem to make sense under these
> > circumstances, simply because all DPCs in the queue have been requested by
> > your NIC’s ISR anyway, so that it just does not matter if the request goes
> > to the head or tail of the queue. By speeding up processing of request X you
> > delay processing of the request Y, which does not seem to make sense if
> > these requests have been originated by the same device.
> >
> > Anton Bassov
> >
> >
> > —
> > Questions? First check the Kernel Driver FAQ at
> > http://www.osronline.com/article.cfm?id=256
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
> >
>
>

5

> When DPC importance is Medium, and DPC is targetted at current CPU or

non-targetted, the Queue-draining interrupt is always called, but when there
are multiple CPUs, and the DPC is targetted at different CPU, when DPC
importance is medium, the interrupt is called only when DPC queue length
exceeds maximum DPC queue length or when system is idle.

Why do you want to target your DPC to some particular processor??? DPC is targeted to the processor that queues it, i.e. the one that runs ISR, by default . The choice of processor to dispense interrupt to is done by the hardware via APIC arbitration protocol ( normally interrupt gets dispensed to the least busy CPU), so that ISR runs on the CPU that is the most appropriate for this task at the moment. If you leave things this way, this scheme insures the most efficient interrupt (and, hence, DPC) handling. Although you can target DPCs to some particular CPUs, you should not do it unless you have very specific needs and are absolutely sure you know what you are doing - otherwise, you are going to get performance degradation, instead of improvement. In fact, your previous post provides an additional proof to my words - according to you, as long as you leave the defaults alone, everything works just fine. Why do you want to change it???

Anton Bassov