Interrupt scheduling

Hello,

My question is about interrupt handling in an NDIS miniport. In my
standard NDIS 5.1 miniport it seems like I am running into a problem of
catching up with h/w interrupts. My driver schedules a packet for
transmission, remembers it and waits for a “transmit complete” interrupt
from h/w. When the interrupt arrives, the driver acknowledges the
interrupt and requests a DPC to run at a later point in time. When the
DPC runs, it completes processing the packet that was sent by NDIS for
transmission.

My driver works okay at lower data rates, but when the packets arrive
quicker (Ethernet 10Mbps speed), it seems like my interrupt handler
doesn’t get scheduled fast enough. My queue fills up and I run out of
resources. My transmission rate is much higher than the 10Mbps rate and
it seems odd that this would happen. I don’t spend a whole lot of time
in each interrupt handler call.

I am trying to understand how long it takes windows to schedule the DPC
before I can really handle the interrupt. Are there ways to speed this
up? I have been looking to see if the sample drivers do anything special
that I might be missing? Do they simply return packets if they run out
of resources?

Thanks
Ravi

One int per packet is not gonna work for modern NIC.
From throughput standpoint, you need to reduce the
number of INTs generated by the NIC because INT is
expensive. That’s why most decent Gbe supports
Interrupt Coalescing AKA Interrupt Moderation.

You need to change the model to N packet per
interrupt, or better, your nic/driver does 1 int per
packet at low throughput (to get low latency)and
enabling Interrupt Coalescing at different level based
on the throughput and CPU usage. Tuning is an art, it
involves alot of research and experiment in driver,
firmware and hardware. Good luck!


Calvin Guan (Windows DDK MVP)
Staff SW Engineer NetXtreme MINIPORT
Broadcom Corp. Irvine, CA
www.broadcom.com

— “Murty, Ravi” wrote:

> Hello,
>
> My question is about interrupt handling in an NDIS
> miniport. In my
> standard NDIS 5.1 miniport it seems like I am
> running into a problem of
> catching up with h/w interrupts. My driver schedules
> a packet for
> transmission, remembers it and waits for a “transmit
> complete” interrupt
> from h/w. When the interrupt arrives, the driver
> acknowledges the
> interrupt and requests a DPC to run at a later point
> in time. When the
> DPC runs, it completes processing the packet that
> was sent by NDIS for
> transmission.
>
> My driver works okay at lower data rates, but when
> the packets arrive
> quicker (Ethernet 10Mbps speed), it seems like my
> interrupt handler
> doesn’t get scheduled fast enough. My queue fills up
> and I run out of
> resources. My transmission rate is much higher than
> the 10Mbps rate and
> it seems odd that this would happen. I don’t spend a
> whole lot of time
> in each interrupt handler call.
>
> I am trying to understand how long it takes windows
> to schedule the DPC
> before I can really handle the interrupt. Are there
> ways to speed this
> up? I have been looking to see if the sample drivers
> do anything special
> that I might be missing? Do they simply return
> packets if they run out
> of resources?
>
> Thanks
> Ravi
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown
> lmsubst tag argument: ‘’
> To unsubscribe send a blank email to
xxxxx@lists.osr.com

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

> [mailto:xxxxx@lists.osr.com]On Behalf Of Calvin Guan

One int per packet is not gonna work for modern NIC.
From throughput standpoint, you need to reduce the
number of INTs generated by the NIC because INT is
expensive. That’s why most decent Gbe supports
Interrupt Coalescing AKA Interrupt Moderation.

One interrupt per packet should be good enough to get
100 megabits/sec on a modern 1-3 GHz CPU.

I would suspect an IRQ sharing with a device that has
a badly written driver that spends too much time in its ISR
or just frequently acknowledges interrupts from another device.

Running the kernel profiler (kernrate.exe) may give
a clue.

Dmitriy Budko VMware

Actually, my description wasn’t accurate. When my interrupt handler gets
invoked, I do handle multiple packets that are completed and keep
looking until I find an entry in the queue that isn’t.

Is there a way to tell when the interrupt handler gets invoked?

Thanks
Ravi

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Calvin Guan
Sent: Tuesday, July 26, 2005 6:15 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Interrupt scheduling

One int per packet is not gonna work for modern NIC.
From throughput standpoint, you need to reduce the
number of INTs generated by the NIC because INT is
expensive. That’s why most decent Gbe supports
Interrupt Coalescing AKA Interrupt Moderation.

You need to change the model to N packet per
interrupt, or better, your nic/driver does 1 int per
packet at low throughput (to get low latency)and
enabling Interrupt Coalescing at different level based
on the throughput and CPU usage. Tuning is an art, it
involves alot of research and experiment in driver,
firmware and hardware. Good luck!


Calvin Guan (Windows DDK MVP)
Staff SW Engineer NetXtreme MINIPORT
Broadcom Corp. Irvine, CA
www.broadcom.com

— “Murty, Ravi” wrote:

> Hello,
>
> My question is about interrupt handling in an NDIS
> miniport. In my
> standard NDIS 5.1 miniport it seems like I am
> running into a problem of
> catching up with h/w interrupts. My driver schedules
> a packet for
> transmission, remembers it and waits for a “transmit
> complete” interrupt
> from h/w. When the interrupt arrives, the driver
> acknowledges the
> interrupt and requests a DPC to run at a later point
> in time. When the
> DPC runs, it completes processing the packet that
> was sent by NDIS for
> transmission.
>
> My driver works okay at lower data rates, but when
> the packets arrive
> quicker (Ethernet 10Mbps speed), it seems like my
> interrupt handler
> doesn’t get scheduled fast enough. My queue fills up
> and I run out of
> resources. My transmission rate is much higher than
> the 10Mbps rate and
> it seems odd that this would happen. I don’t spend a
> whole lot of time
> in each interrupt handler call.
>
> I am trying to understand how long it takes windows
> to schedule the DPC
> before I can really handle the interrupt. Are there
> ways to speed this
> up? I have been looking to see if the sample drivers
> do anything special
> that I might be missing? Do they simply return
> packets if they run out
> of resources?
>
> Thanks
> Ravi
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown
> lmsubst tag argument: ‘’
> To unsubscribe send a blank email to
xxxxx@lists.osr.com

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@intel.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Another related question. Is there a way to tell how fast packets arrive
in the SendPackets entry point in my driver. I want to make sure I don’t
receive packets fast than I can send them. If I put some timers and run
it in the debugger to see the result, doesn’t that slow down the system
and give me incorrect times?

Thanks
Ravi

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Murty, Ravi
Sent: Wednesday, July 27, 2005 7:56 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Interrupt scheduling

Actually, my description wasn’t accurate. When my interrupt handler gets
invoked, I do handle multiple packets that are completed and keep
looking until I find an entry in the queue that isn’t.

Is there a way to tell when the interrupt handler gets invoked?

Thanks
Ravi

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Calvin Guan
Sent: Tuesday, July 26, 2005 6:15 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Interrupt scheduling

One int per packet is not gonna work for modern NIC.
From throughput standpoint, you need to reduce the
number of INTs generated by the NIC because INT is
expensive. That’s why most decent Gbe supports
Interrupt Coalescing AKA Interrupt Moderation.

You need to change the model to N packet per
interrupt, or better, your nic/driver does 1 int per
packet at low throughput (to get low latency)and
enabling Interrupt Coalescing at different level based
on the throughput and CPU usage. Tuning is an art, it
involves alot of research and experiment in driver,
firmware and hardware. Good luck!


Calvin Guan (Windows DDK MVP)
Staff SW Engineer NetXtreme MINIPORT
Broadcom Corp. Irvine, CA
www.broadcom.com

— “Murty, Ravi” wrote:

> Hello,
>
> My question is about interrupt handling in an NDIS
> miniport. In my
> standard NDIS 5.1 miniport it seems like I am
> running into a problem of
> catching up with h/w interrupts. My driver schedules
> a packet for
> transmission, remembers it and waits for a “transmit
> complete” interrupt
> from h/w. When the interrupt arrives, the driver
> acknowledges the
> interrupt and requests a DPC to run at a later point
> in time. When the
> DPC runs, it completes processing the packet that
> was sent by NDIS for
> transmission.
>
> My driver works okay at lower data rates, but when
> the packets arrive
> quicker (Ethernet 10Mbps speed), it seems like my
> interrupt handler
> doesn’t get scheduled fast enough. My queue fills up
> and I run out of
> resources. My transmission rate is much higher than
> the 10Mbps rate and
> it seems odd that this would happen. I don’t spend a
> whole lot of time
> in each interrupt handler call.
>
> I am trying to understand how long it takes windows
> to schedule the DPC
> before I can really handle the interrupt. Are there
> ways to speed this
> up? I have been looking to see if the sample drivers
> do anything special
> that I might be missing? Do they simply return
> packets if they run out
> of resources?
>
> Thanks
> Ravi
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown
> lmsubst tag argument: ‘’
> To unsubscribe send a blank email to
xxxxx@lists.osr.com

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@intel.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com