Cullen,
You might also try the following:
Instead of returning NDIS_STATUS_PENDING from NdisSend() and holding onto
the packet given to you, define some small number of buffers into which you
will queue packets, say, two for starters. When MiniportSend() is called,
if you have a free buffer, copy the packet to the buffer and return
NDIS_STATUS_SUCCESS. If you don’t have any buffers, return
NDIS_STATUS_RESOURCES. NDIS should hold off sending you any more packets
until you call NdisMSendResourcesAvailable() which you should do *instead*
of using NdisMSendComplete().
You could also do this with a single buffer, of course, and there were NICs
that did more or less just this thing. Keep in mind that most of the NDIS
world left serialized miniports behind a long time ago and probably have not
had to think about it for a long time (save for the occasional NDIS WAN
Miniport here & there) so we don’t exactly have this stuff at the top of the
heap!
It just might be that NDIS will be happy enough with this situation instead
of holding that packet for a long time and it will forego whacking your
MiniportReset(). On the other hand, it might still be upset about a large
backlog of send packets in the (hidden) Miniport Send Queue and still think
your NIC has gone off the rails. I don’t know but it may be worth a shot.
Replacing the ‘completion’ handling with a call to
NdisMSendResourcesAvailable() in the single packet buffer case is probably
not that big a deal to code in.
Just a thought.
Good Luck,
-dave
David R. Cattley
Consulting Engineer
Systems Software Development
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Cullen
Sent: Thursday, October 25, 2007 6:30 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] NdisReset criteria
Dear Mr. Dave,
David R. Cattley wrote:
I appologize for the reference to the DDKs. It is hard to know when
that is appropriate and by no means did I mean to offend.
No need to apologize. While I do not post frequently, I do follow this
thread, and I have seen situations where pasting a quote from the docs was
necessary. I was just trying to let you know that I have the full docs
installed. I guess I could have phrased it more gently, but I have a
co-worker who tells me I would benefit from enrolling in a charm school.
The only honest answer I can give you to your specific question is “I
don’t know precisely all of the criteria that NDIS uses to gate if and
when it will call MiniportReset()”. AFAIK, if the duration that the
current send has been pending or the current request has been pending
exceeds the nominal 4-second timeout,
NDIS will whack your MiniportReset(). If there are other cases,
then, I have not seen them occur.
The ‘X’ you are looking for is documented in the DDK in the comments
for MniportCheckForHang() as “… around four seconds”
as well as the only two conditions I am aware of that are measured
against that time period; Request Completion and Send Completion. I
recognize that that is not a satisfactory answer from me or the DDK
documents.
Well, I guess I will have to settle for that.
That said, I can offer some ideas for you to investigate and when you
figure it out, please close the loop here so we can learn something
too.
Sounds like a plan.
As a possible experiment, have you tried running your scenario with
the checked version of NDIS.SYS with a debugger attached and the ‘NDIS
Verifier’, Debug Level, and Debug Areas enabled
verbosely? It may well be that NDIS will spit out ‘reason’ for
deciding to call the MiniportReset() entry.
I will do that tomorrow morning. Unfortunately, it’s too late in the day
for me to get knee deep in experiments. At this time of day, I’ll spend
more time making mistakes than making progress.
Continuing on the ‘taking a long time’ path: You say that the
hardware is ‘slow’ to send packets (I think you mentioned that it was
PIO based). I re-read the posts but did not grok from them an actual
quantitative measure of ‘slow’, especially relative to the reported
link speed. Perhaps I missed it (sorry).
The hardware is slow in that packets are sent to the device one at a time
using PIO over an ISA bus. Also, the device needs an extra wait state
during each PIO write. To send a packet, the driver has to wait for the
device to indicate that hardware resources are available for a send. There
is no DMA or other facility that would allow faster communication with the
hardware. The hardware resources are limited, but even if the device could
buffer many packets, we still have to send them using slow ISA PIO with an
extra wait state.
Our TCP throughput using Iperf is above 5.5 Mbps when we only have one
destination (no duplication of packets). When we have two destinations, the
throughput drops to 2.25 Mbps.
UDP throughput using Iperf is 11 Mbps. We do not duplicate UDP packets, so
that number is the same for any number of destinations.
If you assume the TCP is sending and receiving using the same medium, then
the 5.5 Mbps indicates that the ACKs are using half the available bandwidth.
With two destinations, we have twice the number of ACKs, and therefore half
of the half.
Is it possible that with single sends the driver/hardware is literally
just keeping up with the combined send & request rate (both are
competing for the same Miniport lock in a serialized driver)?
It is entirely possible. I will dig around and see if there are a couple of
signal lines I can toggle when sends and receives occur.
Does NDIS call your MiniportReset() when you contrive the scenario to
*not* be sending packets? For instance, if you unbind all protocols
from the adapter? In that scenario you would only have requests that
could be backing up or some yet to be understood criteria to trigger
the reset. By eliminating the send activity, you could perhaps intuit
that the send path *is* the issue.
We duplicate packets when we have two destinations. If I use the same
driver binary and create a setup with only one destination, we only send one
packet, and the resets do not occur. So there is something about the
duplicate sends that makes NDIS unhappy.
You noted that CPU utilization reported by Taskman does not exceed
60%. How does that compare with the ‘non-duplicating’
driver? That difference would give you a rough estimate of the CPU
cost of sending a packet. That cost should reasonably be almost
‘noise’ in the measurement.
CPU usage is pinned at 60% for both versions of the driver. The hardware
engineer believes that the ISA bus speed is the limiter here, and that the
CPU spends a lot of time waiting for the PIO to finish. He has scoped this
out as best as he can with the test equipment we have available.
I don’t believe (but I can’t say that I know specifically) that NDIS
itself cares about the link data rate and that it uses it to scale any
timeouts. Bound protocols, however, are encouraged to be aware of the
link datarate and to *not* pound send data into the link such that a
large backlog occurs. You might try reducing the values reported in
OID_GEN_TRANSMIT_BUFFER_SPACE and OID_GEN_TRANSMIT_BLOCK_SIZE to
effectively say that the driver can send only a single packet at a
time. Protocols that are nice enough to ask about these operational
characteristics might back down on the rate of packet transmission.
We report OID_GEN_TRANSMIT_BUFFER_SPACE as 0x2000. I won’t embarass the
original coder here by posting his comment, but it indicates that the value
is arbitrary. We report OID_GEN_TRANSMIT_BLOCK_SIZE as 1514, which is the
MTU plus the Ethernet II header size.
Please keep in mind that we are your coleagues here, not oracles.
Sometime the best we can do is share with you our experiences and help
you find a path to the answer. Sure, sometimes (others more
frequently than me for sure!) there is the one word, line, or
paragraph answer and that is that. Honestly, we are trying to help.
You might consider that there might not actually be an answer to your
question but that does not mean there is not a
solution to your problem. NDIS was created by many people over
many years and has resiliently adapted to new requirements while
maintaining backward compatiblity over a remarkable number releases.
It may well be that no person can answer your question in the terms
you have asked it. The Miniport guts have been ‘improved’ in fits and
starts by a few people and even if they sat down and read through the
source code, it might not reveal a documentable answer
(unfortunately). You of course always have the option of calling MSFT
Product Support, opening a case with them, and having someone with
that access try and answer your question.
Once again, the need for charm school seems to have affected my
communication.
I thank everyone who has responded for their time and patience.
There may not be an answer to my specific question, but it appears that I
still have some options for solving the problem thanks to you and everyone
else on this list.
Good Luck,
Thank you, I will probably need it at this rate.
P.S. Mr. Cattley is fine, by the way. I spoke with him tonight.
He & Mom are watching the Red Sox. Me, I go by just Dave 
lol! I know the interweb is breaking down barriers to communication, but I
sometimes feel that manners are being trampled as a side effect. Also, I
grew up in the South, and I was taught to address people more formally than
is the general rule these days. I apologize for any discomfort.
Thank you very much,
Cullen
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer