Delayed NDIS receive indications

NTDEV
1

I have an NDIS intermediate driver running on Win 2K and XP. The driver is fully functional and operational, but I now need to add additional functionality to the driver which seems to require that I indicate received packets to NDIS outside of the receive handler or transfer data complete handlers of the driver. Specifically, packets may arrive compressed which have been fragmented by the remote sender. These fragments are queued (since the entire packet must be present for the driver to perform decompression) and the receive handler returns without indicating the received packet. At some later point in time, after the packet has been re-assembled and decompressed, I call NdisMIndicateReceivePacket(…) (in the context of my receive complete handler) with the decompressed packet in order to indicate the packet up to all bound protocols above the driver.

Using Network Monitor, I can see the reassembled/decompressed packets arriving and they appear valid. However, it appears that the packets are not being accepted by something above my driver - namely, TCP returns ACKs for the packet received prior to the reassembled packet being received and the connection will eventually fail. Occasionally (due to the nature of the compression algorithm), the retransmitted packet will not need fragmentation and the packet is accepted. By comparing the successfully accepted and the not successfully accepted packets, the data in the not accepted packet is correct, and all headers (Ethernet, IP, TCP) are correct (including checksums).

My questions:

1.) Can I use this approach of delaying indicating the receive?

2.) If the answer is Yes, what tools are available other than Network Monitor to debug why TCPIP.SYS (???) apparently does not accept the indicated packet?

3.) If the answer is No, what other methods are available to achieve this functionality?

Thanks in advance,

Ed Lau

ArteraGroup, Inc.
900 Straits Tpke
Middlebury, CT 06762

2

Ed,

There could be a variety of reasons that the reassembled/decompressed packets are ignored.

One would be that “The first buffer in the packet must contain the amount of data specified by the lookahead setting plus the data size of the MAC header”. If this isn’t setup properly, then the packet will be tossed by some protocols (notably TCP/IP).

Another might be handling of NDIS Task Offload. The original fragmented/compressed packet OOB data may be correct for the higher-level protocol, but the reassembled/decompressed packets that are actually indicated may not be correct. See NDIS.com page http://ndis.com/pcakb/KB05280101.htm.

Good luck,

Thomas F. Divine
www.pcausa.com

“Ed Lau” wrote in message news:xxxxx@ntdev…
I have an NDIS intermediate driver running on Win 2K and XP. The driver is fully functional and operational, but I now need to add additional functionality to the driver which seems to require that I indicate received packets to NDIS outside of the receive handler or transfer data complete handlers of the driver. Specifically, packets may arrive compressed which have been fragmented by the remote sender. These fragments are queued (since the entire packet must be present for the driver to perform decompression) and the receive handler returns without indicating the received packet. At some later point in time, after the packet has been re-assembled and decompressed, I call NdisMIndicateReceivePacket(…) (in the context of my receive complete handler) with the decompressed packet in order to indicate the packet up to all bound protocols above the driver.

Using Network Monitor, I can see the reassembled/decompressed packets arriving and they appear valid. However, it appears that the packets are not being accepted by something above my driver - namely, TCP returns ACKs for the packet received prior to the reassembled packet being received and the connection will eventually fail. Occasionally (due to the nature of the compression algorithm), the retransmitted packet will not need fragmentation and the packet is accepted. By comparing the successfully accepted and the not successfully accepted packets, the data in the not accepted packet is correct, and all headers (Ethernet, IP, TCP) are correct (including checksums).

My questions:

1.) Can I use this approach of delaying indicating the receive?

2.) If the answer is Yes, what tools are available other than Network Monitor to debug why TCPIP.SYS (???) apparently does not accept the indicated packet?

3.) If the answer is No, what other methods are available to achieve this functionality?

Thanks in advance,

Ed Lau

ArteraGroup, Inc.
900 Straits Tpke
Middlebury, CT 06762

3

On Tue, 2003-12-09 at 14:33, Ed Lau wrote:

I have an NDIS intermediate driver running on Win 2K and XP. The
driver is fully functional and operational, but I now need to add
additional functionality to the driver which seems to require that I
indicate received packets to NDIS outside of the receive handler or
transfer data complete handlers of the driver. Specifically, packets
may arrive compressed which have been fragmented by the remote sender.
These fragments are queued (since the entire packet must be present
for the driver to perform decompression) and the receive handler
returns without indicating the received packet. At some later point in
time, after the packet has been re-assembled and decompressed, I call
NdisMIndicateReceivePacket(…) (in the context of my receive complete
handler) with the decompressed packet in order to indicate the packet
up to all bound protocols above the driver.

If you’re reassembling the packet in your miniport, you have to make
sure you adjust the header to not say that the packet is fragmented. If
you’re re-using the IP header from the first packet, for example, you’ll
have the MF bit set, and you don’t want that.

being received and the connection will eventually fail. Occasionally
(due to the nature of the compression algorithm), the retransmitted
packet will not need fragmentation and the packet is accepted.

I’m curious to know why most of your packets get fragmented. This is
certainly possible but will result in poor network performance. Are you
crossing a small-MTU link or something? Does your compression alg
occasionally increase the packet size?

By comparing the successfully accepted and the not successfully
accepted packets, the data in the not accepted packet is correct, and
all headers (Ethernet, IP, TCP) are correct (including checksums).

You’ll have to re-calculate the IP header checksum after you change the
flags field around, by the way.

My questions:

1.) Can I use this approach of delaying indicating the receive?

Seems fine to me.

2.) If the answer is Yes, what tools are available other than Network
Monitor to debug why TCPIP.SYS (???) apparently does not accept the
indicated packet?

In WinDbg, load symbols for your target’s tcpip.sys and do x
tcpip!*debug* - you should see some debug variables that you can use to
turn on tcpip.sys debug spew. I’ve used these before with good success.

HTH.

-sd

4

Steve -

The IP header is correct; it is re-built at the receiving side based on the original (i.e., pre-fragmented) header. The checksums of both the IP and TCP header are re-calculated as needed, and are correct as far as Network Monitor is concerned.

Packets do not require fragmentation often, it is only a few (not most) of the packets that encounter this problem - but the failure of the re-assembled packets to be accepted by drivers above mine does indeed kill performance (when it doesn’t actually stop everything in it’s tracks).

Thanks for the advice regarding WinDbg; I usually use SoftICE but may reconsider for the time being. I am attempting now to get the checked version of NDIS.SYS loaded (so that I can enable it’s debug output), but the SP4 (Win 2000) version seems to crash my system during boot (and the SoftICE display is garbled at this point, making debugging quite difficult!).

Thanks again,

Ed Lau

ArteraGroup, Inc.
900 Straits Tpke
Middlebury, CT 06762

----- Original Message -----
From: Steve Dispensa
To: Windows System Software Devs Interest List
Sent: Wednesday, December 10, 2003 11:33 AM
Subject: [ntdev] Re: Delayed NDIS receive indications

On Tue, 2003-12-09 at 14:33, Ed Lau wrote:

I have an NDIS intermediate driver running on Win 2K and XP. The
driver is fully functional and operational, but I now need to add
additional functionality to the driver which seems to require that I
indicate received packets to NDIS outside of the receive handler or
transfer data complete handlers of the driver. Specifically, packets
may arrive compressed which have been fragmented by the remote sender.
These fragments are queued (since the entire packet must be present
for the driver to perform decompression) and the receive handler
returns without indicating the received packet. At some later point in
time, after the packet has been re-assembled and decompressed, I call
NdisMIndicateReceivePacket(…) (in the context of my receive complete
handler) with the decompressed packet in order to indicate the packet
up to all bound protocols above the driver.

If you’re reassembling the packet in your miniport, you have to make
sure you adjust the header to not say that the packet is fragmented. If
you’re re-using the IP header from the first packet, for example, you’ll
have the MF bit set, and you don’t want that.

being received and the connection will eventually fail. Occasionally
(due to the nature of the compression algorithm), the retransmitted
packet will not need fragmentation and the packet is accepted.

I’m curious to know why most of your packets get fragmented. This is
certainly possible but will result in poor network performance. Are you
crossing a small-MTU link or something? Does your compression alg
occasionally increase the packet size?

By comparing the successfully accepted and the not successfully
accepted packets, the data in the not accepted packet is correct, and
all headers (Ethernet, IP, TCP) are correct (including checksums).

You’ll have to re-calculate the IP header checksum after you change the
flags field around, by the way.

My questions:

1.) Can I use this approach of delaying indicating the receive?

Seems fine to me.

2.) If the answer is Yes, what tools are available other than Network
Monitor to debug why TCPIP.SYS (???) apparently does not accept the
indicated packet?

In WinDbg, load symbols for your target’s tcpip.sys and do x
tcpip!*debug* - you should see some debug variables that you can use to
turn on tcpip.sys debug spew. I’ve used these before with good success.

HTH.

-sd

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@midcore.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

5

On Wed, 2003-12-10 at 13:31, Ed Lau wrote:

Thanks for the advice regarding WinDbg; I usually use SoftICE but may
reconsider for the time being. I am attempting now to get the checked
version of NDIS.SYS loaded (so that I can enable it’s debug output),
but the SP4 (Win 2000) version seems to crash my system during boot
(and the SoftICE display is garbled at this point, making debugging
quite difficult!).

I’m sure there’s a way to do this in softice too; I just don’t know it.

Checked NDIS.SYS asserting is probably a bad sign, though. I don’t know
how softice handles asserts, but I’d start your search there.

Also, now is probably the time to set up the NDIS verifier. There’s
lots of documentation about this on the web.

You might also add a checked tcpip.sys - i think the free build lacks
the debug output I mentioned earlier.

-sd

6

Have you observed any pattern in the rejected packets? Are you sure that NetMon
wasn’t unhappy with them specifically? It sounds like you’ve got things 99%
correct and are failing in some specific conditions.

Ed Lau wrote:

The IP header is correct; it is re-built at the receiving side based on the
original (i.e., pre-fragmented) header. The checksums of both the IP and TCP
header are re-calculated as needed, and are correct as far as Network Monitor
is concerned. Packets do not require fragmentation often, it is only a few
(not most) of the packets that encounter this problem - but the failure of the
re-assembled packets to be accepted by drivers above mine does indeed kill
performance (when it doesn’t actually stop everything in it’s tracks).

–
If replying by e-mail, please remove “nospam.” from the address.

James Antognini
Windows DDK MVP

7

> ----------

From:
xxxxx@positivenetworks.net[SMTP:xxxxx@positivenetworks.net]
Reply To: xxxxx@lists.osr.com
Sent: Wednesday, December 10, 2003 8:45 PM
To: xxxxx@lists.osr.com
Subject: [ntdev] Re: Delayed NDIS receive indications

On Wed, 2003-12-10 at 13:31, Ed Lau wrote:
> Thanks for the advice regarding WinDbg; I usually use SoftICE but may
> reconsider for the time being. I am attempting now to get the checked
> version of NDIS.SYS loaded (so that I can enable it’s debug output),
> but the SP4 (Win 2000) version seems to crash my system during boot
> (and the SoftICE display is garbled at this point, making debugging
> quite difficult!).

I’m sure there’s a way to do this in softice too; I just don’t know it.

Yes, it is possible and almost the same. Load correct symbols and then
change variables.

Checked NDIS.SYS asserting is probably a bad sign, though. I don’t know
how softice handles asserts, but I’d start your search there.

It should catch RtlAssert, display failed assertion and stop allowing debug
or continue. OP should make sure correct symbols are loaded and up-to-date
SoftICE used. I guess for w2k SP4 DS 3.01 is necessary but I’m not sure.
NuMega tech support will know better.

Also, now is probably the time to set up the NDIS verifier. There’s
lots of documentation about this on the web.

You might also add a checked tcpip.sys - i think the free build lacks
the debug output I mentioned earlier.

It is a good idea. Yet another possibility I used successfully – set
breakpoints on tcpip.sys protocol handlers (_ArpXxx) and when changed packet
is indicated, step through code until packet is refused. For simple bugs as
bad checksum it was the fastest way (usually 1 - 2 minutes to see a reason).

Best regards,

Michal Vodicka
STMicroelectronics Design and Application s.r.o.
[michal.vodicka@st.com, http:://www.st.com]

8

I can’t help you with your NDIS problem, but I can help you get setup so
that the SoftICE screen is not garbled.

With Win2k SP4 Microsoft broke (I’m assuming) the vga code and are leaving
some vga controllers in an odd state. This is what is causing the garbled
screen that you are seeing.

To work around it you will need to do the following:
(1) Using Regedit (insert all the usual warning that come along with using
regedit here) go to HKLM\System\CurrentControlSet\Services\Siwvid and change
its Group to be “Boot Bus Extender” without the quotes.
(2) Make note of its tag value, it should be 0x10.
(3) Go to the following registry key,
HKLM\System\CurrentControlSet\Control\GroupOrderList.
(4) Double click on boot bus extender. Increase the first byte by one.
After entering in the +1 value you will most likely need to hit the delete
key depending on which registry editor you are using.
(5) Type in the dword of the Siwvid tag. Should be hex ten. You would
enter this as (assuming byte display is showing) 00 00 00 10
(6) Make sure /noguiboot is in your boot.ini file for the os that is being
debugged
(7) Reboot.

At this point the SoftICE screen should show up fine. If you are unsure or
have any problems please contact our support department.

Chris
Chris Plakyda
xxxxx@compuware.com mailto:xxxxx
SoftICE Manager
Tech Lead SoftICE
Development Lead SoftICE
DriverStudio Software Engineer
Numega/Compuware
1-603-578-8289

The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose
it to anyone else. If you received it in error please notify us immediately
and then destroy it.</mailto:xxxxx>

9

> I have an NDIS intermediate driver running on Win 2K and XP

Have you verified that the packet you’re creating that will hold the
re-assembled data does not have a total length that exceeds the MTU of the
miniport that it was indicated from? I can’t speak with authority on this
one as to whether or not it will cause problems as I’ve not confirmed it,
but my hunch is that it will. If you are indeed doing this I’m sure someone
else on the list will be able to say whether or not this will cause a
problem.

At some later point in time, after the packet has been re-assembled and
decompressed,
I call NdisMIndicateReceivePacket(…) (in the context of my receive
complete handler)
with the decompressed packet in order to indicate the packet up to all
bound protocols
above the driver.

When you do this, do you set the header length accordingly
(NDIS_SET_PACKET_HEADER_SIZE()) on the NDIS_PACKET’s that will be indicated?
I’m assuming that you are creating new NDIS_PACKET’s. Also, (as Thomas
Divine suggested) you must ensure that the first buffer in the chain is at
least MAC header length + lookahead length, else bad things may happen.

The times I’ve run into indication problems in the past have been mainly due
to checksum issues, but as you’ve said this isn’t the problem.

Matt

10

Steve -

Thanks for bringing up NDIS verifier (I assume you mean NDISTest ?). I had forgotten about that. The assertions in NDIS.SYS, it turns out (after running the checked version on another PC) were assertions thrown due to no TransferComplete handler specified by NDISWAN.SYS. The checked version didn’t help in identifying a problem, although I admit that I didn’t actually step through the NdisMIndicateReceivePacket code. I’m going to go ahead and install the checked version of TCPIP.SYS; does anyone know if it contains debug tracing capabilities similar to NDIS.SYS? I couldn’t find any information in Microsoft’s KB about TCPIP.SYS debug output specifically …

Thanks again,

Ed Lau

ArteraGroup, Inc.
900 Straits Tpke
Middlebury, CT 06762

----- Original Message -----
From: Steve Dispensa
To: Windows System Software Devs Interest List
Sent: Wednesday, December 10, 2003 2:45 PM
Subject: [ntdev] Re: Delayed NDIS receive indications

On Wed, 2003-12-10 at 13:31, Ed Lau wrote:

Thanks for the advice regarding WinDbg; I usually use SoftICE but may
reconsider for the time being. I am attempting now to get the checked
version of NDIS.SYS loaded (so that I can enable it’s debug output),
but the SP4 (Win 2000) version seems to crash my system during boot
(and the SoftICE display is garbled at this point, making debugging
quite difficult!).

I’m sure there’s a way to do this in softice too; I just don’t know it.

Checked NDIS.SYS asserting is probably a bad sign, though. I don’t know
how softice handles asserts, but I’d start your search there.

Also, now is probably the time to set up the NDIS verifier. There’s
lots of documentation about this on the web.

You might also add a checked tcpip.sys - i think the free build lacks
the debug output I mentioned earlier.

-sd

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@midcore.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

11

Unfortunately, the only difference I see between accepted and rejected packets are that the rejected packets are indicated outside of the context of receive handler where the packet was originally received by my driver, and obviously the IP identification numbers are different for packets that have been retransmitted by the server to my driver (and hence the IP checksum as well). Netmon does not complain about any of the rejected packets. Comparing two packets, one a rejected reassembled packet, the other an accepted non-fragmented packet, shows that they are identical (identical as far as what is logged by Netmon) with the above exceptions - IP Ident and IP Checksum. This driver has been in use for several years performing NAT; the main difference is that the code path is “modify and then send” (to another PC, NAT doesn’t make sense on a single PC), whereas in this case I am trying to “modify and indicate up the stack” on the local PC. This is different in the case of the NAT portion of the code where packets are indicated up the stack in that the indication is performed at a later date, after the driver has left the scope of the receive handler. I assume that the root cause will be something simple that I have missed; the real dilemma is in identifying what simple thing that might be.

Thanks,

Ed Lau

ArteraGroup, Inc.
900 Straits Tpke
Middlebury, CT 06762

----- Original Message -----
From: James Antognini
Newsgroups: ntdev
To: Windows System Software Devs Interest List
Sent: Wednesday, December 10, 2003 3:21 PM
Subject: [ntdev] Re: Delayed NDIS receive indications

Have you observed any pattern in the rejected packets? Are you sure that NetMon
wasn’t unhappy with them specifically? It sounds like you’ve got things 99%
correct and are failing in some specific conditions.

Ed Lau wrote:

The IP header is correct; it is re-built at the receiving side based on the
original (i.e., pre-fragmented) header. The checksums of both the IP and TCP
header are re-calculated as needed, and are correct as far as Network Monitor
is concerned. Packets do not require fragmentation often, it is only a few
(not most) of the packets that encounter this problem - but the failure of the
re-assembled packets to be accepted by drivers above mine does indeed kill
performance (when it doesn’t actually stop everything in it’s tracks).

–
If replying by e-mail, please remove “nospam.” from the address.

James Antognini
Windows DDK MVP

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@midcore.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

12

Thanks everyone for all your help.

It turns out that the problem was what Thomas had originally suggested (and Matt reminded me of). I had been chaining a header buffer (of the size of the header) and a separate buffer with the data, rather than a header buffer that contained the header plus enough data to satisfy the requirement that the first buffer is the size of the header + the indicated lookahead size. I looked in the Win 2000 DDK documentation and it does not mention this requirement; I’m happy to see that it is mentioned in the Win XP DDK.

Thanks again everyone,

Ed Lau

ArteraGroup, Inc.
900 Straits Tpke
Middlebury, CT 06762

----- Original Message -----
From: Matt Miller
To: Windows System Software Devs Interest List
Sent: Wednesday, December 10, 2003 3:49 PM
Subject: [ntdev] Re: Delayed NDIS receive indications

I have an NDIS intermediate driver running on Win 2K and XP

Have you verified that the packet you’re creating that will hold the
re-assembled data does not have a total length that exceeds the MTU of the
miniport that it was indicated from? I can’t speak with authority on this
one as to whether or not it will cause problems as I’ve not confirmed it,
but my hunch is that it will. If you are indeed doing this I’m sure someone
else on the list will be able to say whether or not this will cause a
problem.

At some later point in time, after the packet has been re-assembled and
decompressed,
I call NdisMIndicateReceivePacket(…) (in the context of my receive
complete handler)
with the decompressed packet in order to indicate the packet up to all
bound protocols
above the driver.

When you do this, do you set the header length accordingly
(NDIS_SET_PACKET_HEADER_SIZE()) on the NDIS_PACKET’s that will be indicated?
I’m assuming that you are creating new NDIS_PACKET’s. Also, (as Thomas
Divine suggested) you must ensure that the first buffer in the chain is at
least MAC header length + lookahead length, else bad things may happen.

The times I’ve run into indication problems in the past have been mainly due
to checksum issues, but as you’ve said this isn’t the problem.

Matt

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@midcore.com
To unsubscribe send a blank email to xxxxx@lists.osr.com