Right, you’ve identified a fundamental race that is impossible to solve. Fortunately, you’re not expected to solve it.
What we really need from you is a best-effort for three things:
- If a single call to MiniportSendNetBufferLists has two packets A1 and A2; you will transmit A1 before transmitting A2.
- If we call you at DISPATCH_LEVEL on CPU X to send packet B1, then later call you again at DISPATCH_LEVEL on CPU X to send packet C1; you will transmit B1 before transmitting C1.
- If we tell you to send packet D1, and you call NdisMSendNetBufferListsComplete(D1), then later we tell you to send packet E1; you will transmit D1 before transmitting E1.
You may break the above rules, if instructed to by QOS protocols like 802.1P, WMM, or DCB. You are allowed to occasionally bend one of the rules in limited corner cases, if you must (e.g., perhaps immediately after resuming from low-power, you flush a queue of packets that could temporarily race with new incoming packets). The purpose of these rules is to try to keep the recipient on the fast-path of in-order packet delivery. The recipient must correctly handle packet re-ordering, but we don’t want to force him to take that slower path in the common case. So if you make a rare reordering once every few hours, that’s not a big deal. But the more you reorder packets, the user’s network applications will get slower.
All three requirements are achievable if your send handler looks something like this:
MiniportSendNetBufferLists(NblsToSend)
{
AcquireLock();
while (NblsToSend)
{
Nbl = PopListHead(NblsToSend);
PushListTail(Adapter->TxQueue);
}
ReleaseLock();
}
Of course, there are ways to optimize this. But the point is, you keep the NBLs roughly ordered within a call, and between calls (on the same CPU). What we *really* don’t want you to do is to something like this:
MiniportSendNetBufferListsBad(NblsToSend)
{
// bad - reorders the NBL chain
InterlockedPushListSList(Adapter->TxQueue, NblsToSend, …);
}
(In general it’s dangerous to use SLISTs for the TX path. It can be made to work, but you have to think carefully about reordering.)
The NDISTest framework is pretty good about letting you know if you’ve reordered NBLs. I think it tolerates some amount of reordering at high throughput, but is stricter on low-throughput tests.
In contrast to the Send and Receive paths, the SendComplete and ReceiveReturn paths do not care about order of NBLs. So feel free to gratuitously alphabetize the NBLs however you want in SendComplete and ReceiveReturn. (NDIS itself will haphazardly reorder packets in these handlers in some cases.)
I’m not making it up though.