NDIS protocol driver receive jitter

Hi,

Our problem is as follows:
The pakets are accepted NDIS protocol driver from changing greatly delayed, which depends on the loading system (and especially from the mouse wheel in IE :).
There is the hardware device that generates packets at regular intervals (~ 200 microseconds).
In the protocol driver packets come without a loss, but at intervals from 0 to 15000 microseconds.

Regards,

Maxim

xxxxx@rambler.ru wrote:

Hi,

Our problem is as follows:
The pakets are accepted NDIS protocol driver from changing greatly delayed, which depends on the loading system (and especially from the mouse wheel in IE :).
There is the hardware device that generates packets at regular intervals (~ 200 microseconds).
In the protocol driver packets come without a loss, but at intervals from 0 to 15000 microseconds.

Windows is not a real time OS.
Let your users decide what is more important to them : IE with mouse
wheel, or something else.

Regards,
–PA

> In the protocol driver packets come without a loss, but at intervals from 0 to 15000 microseconds.

Is this delay stable? how is the TCP connection throughput affected by this?

If not affected at all, i.e. delays are statistically random and not stable - then forget them. Network is not a realtime device, a single packet can curely be delayed 15ms, this is OK. What is not OK if the connection bandwidth suffers.

So, just measure the raw TCP throughput with your filter and without it. If the throughput will be the same - then mark this issue as “Closed” in your bugtracker :slight_smile: the exact packet arrival times are not guaranteed anyway.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

Maxim S. Shatskih wrote:

> In the protocol driver packets come without a loss, but at intervals from 0 to 15000 microseconds.

Is this delay stable? how is the TCP connection throughput affected by this?

If not affected at all, i.e. delays are statistically random and not stable - then forget them. Network is not a realtime device, a single packet can curely be delayed 15ms, this is OK. What is not OK if the connection bandwidth suffers.

So, just measure the raw TCP throughput with your filter and without it. If the throughput will be the same - then mark this issue as “Closed” in your bugtracker :slight_smile: the exact packet arrival times are not guaranteed anyway.

Hmm. 15 ms is the default timer resolution on newer PCs (APIC).

–pa

> Hmm. 15 ms is the default timer resolution on newer PCs (APIC).

Correct, looks like his packet handling gets rescheduled.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

As I pointed out the delays occurring in the protocol driver. And in our assessment, they had nothing to do with the TCP traffic.
Load can be created and not IE, almost any program related to the GUI. For example Visual Studio help.
Another observation:

  • On multiprocessor machines, the maximum delay is higher than for single;
  • If on multiprocessor machines set the registry flag for NDIS ProcessorAffinityMask = 0x01
    it significantly reduces delays.

We have assumed that this could be related to that:
"Only one driver shipped with NT, NDIS, uses DPC targeting and priorities.
NDIS uses low importance DPCs that are targeted at a particular processor for its DPC processing. "
http://technet.microsoft.com/en-us/sysinternals/bb963898.aspx

Of course, the assumption is a bit artificial. But is unclear as can affect any user mode load on the thread that ran on the DPC level.

> As I pointed out the delays occurring in the protocol driver. And in our assessment, they had nothing

to do with the TCP traffic.

What are measurement figures in cases of:

  • TCP connection bandwidth to the neighbouring machine, measured using a simple tiny socket-based C app, measures without the delaying load from GUI?
  • same TCP connection bandwidth measured by the same app and with the same machine, but with the delaying load?
  • same 2 cases for UDP
  • Round Trip Time to a neighbouring machine with and without the delaying load.

First measure these things. Then make a decision of whether the delay in your IM is actually important or can be just plain ignored.

Ethernet network is not a realtime device. So, 15ms of “non-realtimeness” does not look too bad. Measure the real things affected by this, and make a decision.

For instance, 15ms delay can be introduced by the interrupt moderation feature of the modern NIC, where, under load, it disables its interrupt and starts to poll the hardware by Windows timer instead, and standard Windows timer resolution on modern machines is around 15ms (1/60s).

So, the NIC accumulates a cluster of packets in its receive DMA and handles it to the processing by the software only once per 15ms, and this is OK.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

>

For instance, 15ms delay can be introduced by the interrupt
moderation

feature of the modern NIC, where, under load, it disables its
interrupt
and starts to poll the hardware by Windows timer instead, and standard
Windows timer resolution on modern machines is around 15ms (1/60s).

So, the NIC accumulates a cluster of packets in its receive DMA and
handles it to the processing by the software only once per 15ms, and
this is OK.

You may have been simplifying to illustrate your point (and my apologies
if you were), but I don’t think that is quite what interrupt moderation
is.

At 1GBit/second you could get up to 1.5Mbytes of data in 15ms, so the
CPU would be doing nothing for 15ms and then suddenly have to process
1.5Mbytes in one hit. That is not OK IMHO.

Interrupt moderation in its simplest form would normally involve telling
the hardware to only interrupt you every x packets instead of every
single packet. So under high load you can cut the interrupt rate by 10
by asking to only be interrupted every 10 packets, while still
maintaining decent latency. You also set a timer to pick up the case
where the load dropped and you suddenly only got 3 packets after 15ms,
so that packets still get processed in reasonable time. Such interrupt
moderation would only turn on under reasonably high load, and the 15ms
is a ‘worst case’ thing where the load drops suddenly, so I wouldn’t
expect to see it very often at all.

High end (or maybe all these days?) NIC’s do it all in hardware where
you can set things like:
. The number of packets before generating an IRQ (eg 20)
. The maximum time to elapse since the last packet before generating an
IRQ (eg 200us since the last packet if we are not up to 20 packets yet)
. The maximum time to elapse since the first packet before generating an
IRQ (eg if packets were coming in every 100us and we wait until 20 then
that would be 2ms which may be unacceptably high)

There are a whole load more knobs to tweak too according to the spec
sheet of current Intel adapters… Fascinating stuff!

James

xxxxx@rambler.ru wrote:

As I pointed out the delays occurring in the protocol driver. And in our assessment, they had nothing to do with the TCP traffic.
Load can be created and not IE, almost any program related to the GUI. For example Visual Studio help.
Another observation:

  • On multiprocessor machines, the maximum delay is higher than for single;
  • If on multiprocessor machines set the registry flag for NDIS ProcessorAffinityMask = 0x01
    it significantly reduces delays.

We have assumed that this could be related to that:
"Only one driver shipped with NT, NDIS, uses DPC targeting and priorities.
NDIS uses low importance DPCs that are targeted at a particular processor for its DPC processing. "
http://technet.microsoft.com/en-us/sysinternals/bb963898.aspx

Of course, the assumption is a bit artificial. But is unclear as can affect any user mode load on the thread that ran on the DPC level.

Why unclear? User mode load causes driver activity: video, disk, sound,
network, input. All these can issue unpredictable number of DPCs, that
can delay your precious TCP packets to unpredictable time.
Or, in short: Windows (as a whole product, incl. all in-box and 3rd
party drivers) is not a real time OS.

–pa

(1) We have already disabled adaptive interrupts on the NIC.
(2) Delays depend on the load of the system (user mode)
(3) Delays can be as high as 15ms (15000 microseconds)

> (1) We have already disabled adaptive interrupts on the NIC.

(2) Delays depend on the load of the system (user mode)
(3) Delays can be as high as 15ms (15000 microseconds)

Once (well, 3rd time) more:

  • what is the effect of these delays on average RTT (ping time)? on TCP connection throughput? on UDP data throughput?

The reason is that probably such a tiny delay of 15ms is unavoidable and is “by design” of the general-purpose networking on a general-purpose OS. It clearly can be the bad-case DPC latency in the presence of lots of other DPCs (from mouse, from sound hardware, from disk drives and so on).

Use ATM hardware with a realtime OS like QNX if you’re offended with this :slight_smile:

The only thing when you will real need to deal with the delay is when it adversely affects things like total network throughput (mentioned above).


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

I have yet to test on Vista, but, on XP and older OS’s I have seen packet delivery to a user mode application from our IM driver to vary greatly with high CPU or high disk usage. I have seen delays up to 30 seconds between the time our IM driver indicates the packet up and when the application actually receives it.