At 10:10 AM 6/1/2000 -0700, Gary Little wrote:
Thanks to all that have taken the time to reply. Your comments and insights
are deeply appreciated. However, can anyone point to some solid
documentation or a white paper on this topic?
I can’t remember the name of the paper, but about 3 years ago Microsoft
produced a paper about interrupt handling and its effect on network
performance, principally applied to their routing software.
The conclusion of the paper was that under heavy load, by switching to
polling the Ethernet card the performance gain was dramatic and also
significantly cut the CPU load. If I remember correctly, this paper had
some interesting figures on interrupt handling and latency. You’ll have to
search the Microsoft archives, I was working in Boston at the time so it
must have been around winter/spring 1996-97.
HTH,
Mark.
The best way to improve “efficiency” would be to reduce the number of
interrupts per second (far to much time burned up in dispatching interrupts
by NT).
So card vendors out there how about implementing the ability to notify
completion of several IO operations in a single interrupt. That way we don’t
have to pole but can still be interrupt driven.
I know of one Fibre Channel vendor that has this ability; maybe you could
use them in your solution (that is if your not using your own HBA). Mr.
Vendor are you listening?
As others have said, I don’t think the number of interrupts per second will
be your only problem. You will have to deal with PCI bus contention, memory
bus contention, ScsiPort synchronization, etc.
-Shawn
Shawn Carl Erickson (805) 883-4319 [Telnet]
Hewlett Packard Company HPSO/SSMO (Santa Barbara)
-----Original Message-----
From: Gary Little [SMTP:xxxxx@delphieng.com]
Sent: Thursday, June 01, 2000 8:14 AM
To: NT Developers Interest List
Subject: Maximum interrupts per second
Somewhere in the course of my travels in developing NT device drivers I
came across one of those magic numbers that represents the maximum number
of interrupts per second: 10,000. Does NT begin governing interrupts once
interrupts exceeds 10K per second, and if so how does this scale across
multiple CPU’s? It seems that we have a customer that wants to use 20 PCI
fibrechannel adapters for a large storage system, and is expecting on the
order of 200,000 interrupts per second. Will NT or Win2K choke when it
only has about 5 microseconds between interrupts?
A couple of years ago I was servicing ~30,000 IPS on a Single PPro 200
on NT4.0 (SP3?).
The issues I had were SMP related. Never try to do high interrupt
rates on SMPs without interrupt affinity. If you are using SCSI miniports,
the only way to do interrupt affinity is to hack up the Hal.
I think the algorithm required a thread to run every once in a while to
do some cleanup, or the events generating the interrupts would stop, so
I didn’t get locked out.
I my case it was also Fibre Channel, and i was getting about 33K I/O ops
per second, which was about 69K interrupt generating events/sec.
Therefore I was averaging ~3 events per interrupt, because the hardware
protocol allowed events to be queued.
-DH
----- Original Message -----
From: “Gary Little”
To: “NT Developers Interest List”
Sent: Thursday, June 01, 2000 11:14 AM
Subject: [ntdev] Maximum interrupts per second
> Somewhere in the course of my travels in developing NT device drivers I came
> across one of those magic numbers that represents the maximum number of
> interrupts per second: 10,000. Does NT begin governing interrupts once
> interrupts exceeds 10K per second, and if so how does this scale across
> multiple CPU’s? It seems that we have a customer that wants to use 20 PCI
> fibrechannel adapters for a large storage system, and is expecting on the
> order of 200,000 interrupts per second. Will NT or Win2K choke when it only
> has about 5 microseconds between interrupts?
>
>
> We redesigned our drivers to do “interrupt throttling”, such that it kept
track of the interrupt rate, and when it went above 1000 interrupts per
Another way of device design to prevent too many interrupts is using a
“device program” kept in the main memory and accessed by the device via
PCI busmastering.
The device program is a chain of “operations”. Each “operation” contains a
pointer (logical DMA address) to the next “operation”, pointer the (logical
DMA address) to the data buffer, operation code, completion status field and
flags field. One of the flag bits is defined as interrupt flag.
The device hardware logic:
- read the first “operation” (the logical address is in the device register
written by the CPU).
- perform the operation - knowing the operation code and data buffer.
- update the completion status field (will be analyzed by the driver in the
future).
- if interrupt flag is on - assert the INTx line (it is held asserted till
the CPU will
access some register) - and immediately continue.
- if the “pointer to next” field is zero - switch to stopped state (till
some CPU
actions).
- else fetch the next “operation” from the address specified by “pointer to
next” field and repeat all of this once more.
Even rather strict realtime requirements can be satisfied by such a design
without using the terrible number of interrupts per second (provided that
the
driver had allocated enough memory for device program and will not allow
this pipeline to drain and the device to switch to stopped state due to end
of
the device program).
For details, look in OHCI1394 hardware spec available from the Internet as a
PDF file (the 1394 controller is a very high-performance device with
realtime
requirements). I don’t think such a design is a bad idea for SCSI, gigabit
Ethernet. or some realtime hardware too.
BTW - one person told me that this was a standard way of peripheral design
for IBM/360 mainframes.
Max