Query about deserialized NDIS miniports

Hello All!

:slight_smile:

I’m a bit confused by the exact implication of the following rule (bullet point #4) about deserialized NDIS miniports as mentioned @

http://msdn.microsoft.com/en-us/library/ff546395(VS.85).aspx

I quote:

“If a deserialized miniport driver cannot immediately complete send requests, it cannot return the requests to NDIS for requeuing. Instead, the miniport driver must queue send requests internally until sufficient resources are available to transmit the data”

Does this mean that the miniport *MUST* maintain its own SW private queueing mechanism (completely independent of the sw+hw TX rings) for NBLs it cannot send out for some transient resource starvation condition(s)?

For example, as per this rule, if my TX ring does not have any free TX hw/sw descriptors for an outgoing TX NBL, I *MUST NOT* complete the send with an error but *MUST* put that NBL in a private queue until such time that the resources (TX hw descs) are available again. Is this the correct interpretation?

If this indeed is the case, I was wondering why this seemingly fundamental rule was not implemented the xframeii miniport sample in the WDK.

Any clarifications on this subject would be highly appreciated.

TIA

:slight_smile:

/dn0

> Does this mean that the miniport *MUST* maintain its own SW private queueing mechanism

Yes, every NDIS6 miniport must implement a TX queue.

why this seemingly fundamental rule was not implemented the xframeii miniport sample

On the contrary – it *is* implemented in xframeii. But don’t feel bad for missing that – xframeii is not famous for being readable code. If you’re new to NDIS driver development, you will likely find xframeii to be hostile and baffling. (I’m *not* new to NDIS driver development, and I still find xframeii hostile and baffling). I suggest starting with netvmini instead. (Ignore all its VMQ stuff).

The NBLs are queued in pFifo->NBLWaitQ. The TX path in the xframeii sample looks something like this:

xenaMpSendNetBufferLists(NBL chain)
for each NBL in chain :
xmpNicTxNbl() :
compute checksum offload, LSO, etc
add NBL to pFifo->NBLWaitQ
xmpNicTxServe() :
while pop next NBL from NBLWaitQ :
if there insufficient TX descriptors :
restore NBL to front of NBLWaitQ
return from xmpNicTxServe
NdisMAllocateNetBufferSGList() :
(NDIS maps the NB)
xmpProcessSGListCB() :
program hardware

(I am worried that the critical whitespace in this diagram will be destroyed… if so, forgive me. Perhaps the OSR web interface to this mailing list will do a better job than my email client?)

If you have a simple Ethernet driver, you don’t want to queue whole NBLs (like xframeii does). Instead, you want to queue NET_BUFFERs. The reason is that you wind up with very tricky forward progress and correctness issues with whole NBL queuing. (What do you do if you can only map the first NET_BUFFER on an NBL?) So your TX path should be more like netvmini, where you take the chain of NBLs and turn it into a queue of NBs to transmit.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@live.com
Sent: Sunday, October 31, 2010 9:05 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Query about deserialized NDIS miniports

Hello All!

:slight_smile:

I’m a bit confused by the exact implication of the following rule (bullet point #4) about deserialized NDIS miniports as mentioned @

http://msdn.microsoft.com/en-us/library/ff546395(VS.85).aspx

I quote:

“If a deserialized miniport driver cannot immediately complete send requests, it cannot return the requests to NDIS for requeuing. Instead, the miniport driver must queue send requests internally until sufficient resources are available to transmit the data”

Does this mean that the miniport *MUST* maintain its own SW private queueing mechanism (completely independent of the sw+hw TX rings) for NBLs it cannot send out for some transient resource starvation condition(s)?

For example, as per this rule, if my TX ring does not have any free TX hw/sw descriptors for an outgoing TX NBL, I *MUST NOT* complete the send with an error but *MUST* put that NBL in a private queue until such time that the resources (TX hw descs) are available again. Is this the correct interpretation?

If this indeed is the case, I was wondering why this seemingly fundamental rule was not implemented the xframeii miniport sample in the WDK.

Any clarifications on this subject would be highly appreciated.

TIA

:slight_smile:

/dn0


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

The driver defines a maximum number of transmit descriptors (it can be set
in the registry). From that it allocates and calculates resources at init
time.

When asked OID_GEN_TRANSMIT_BUFFER_SPACE it uses that value minus the number
of descriptors in use to tell NDIS or another entity how many more Tx NBLs
can be sent before it will just throw them away.

The driver does implement internal queuing for packets it cannot send
‘immediately’ - but with a limit.

Or at least that is how I read it.

Basically, the number of packets that can be ‘buffered’ pending send is
derived from the value of OID_GEN_TRANSMIT_BUFFER_SPACE /
OID_GEN_TRANSMIT_BLOCK_SIZE.

Good Luck,
Dave Cattley

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@live.com
Sent: Monday, November 01, 2010 12:05 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Query about deserialized NDIS miniports

Hello All!

:slight_smile:

I’m a bit confused by the exact implication of the following rule (bullet
point #4) about deserialized NDIS miniports as mentioned @

http://msdn.microsoft.com/en-us/library/ff546395(VS.85).aspx

I quote:

“If a deserialized miniport driver cannot immediately complete send
requests, it cannot return the requests to NDIS for requeuing. Instead, the
miniport driver must queue send requests internally until sufficient
resources are available to transmit the data”

Does this mean that the miniport *MUST* maintain its own SW private queueing
mechanism (completely independent of the sw+hw TX rings) for NBLs it cannot
send out for some transient resource starvation condition(s)?

For example, as per this rule, if my TX ring does not have any free TX hw/sw
descriptors for an outgoing TX NBL, I *MUST NOT* complete the send with an
error but *MUST* put that NBL in a private queue until such time that the
resources (TX hw descs) are available again. Is this the correct
interpretation?

If this indeed is the case, I was wondering why this seemingly fundamental
rule was not implemented the xframeii miniport sample in the WDK.

Any clarifications on this subject would be highly appreciated.

TIA

:slight_smile:

/dn0


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Hey Jeffrey

Many thanks for your quick response :slight_smile:

Umm… Then why does NicTxNbl() return (and complete) a TX NBL with an error in lines 2078 - 2085 ?

XF_Q_DEL_HEAD(&pFifo->NBLCtxQ, link);
if ( !link )
{ /* out of NBL contexts, can’t even queue this NBL. */
XF_DPR_FREE_SLOCK(&pFifo->TxLock);
Status = NDIS_STATUS_RESOURCES;
pFifo->Stats.TxNBLUnder++;
/* XMPTRACE(XMP_DBG_WRN, (“xmpNicTxNBL:Out of NBL Contexts\n”)); */
/* ASSERT(FALSE); */
break;
}

BTW, the actual send complete with error is in the same fn @ lines 2111 - 2115… ‘Serve’ is not set, so xmpNicTxServe() is not called. Is this then a bug in the sample code?

Yeah good point. But that’s not, er, a design that you should strive to emulate. The xframeii driver is trying to associate its “NBL context” (xmpTxNBLCtx_t) with each NBL *before* queuing, but this is troublesome for exactly the reason you point out. I would pretend that this “NBL context” doesn’t exist; it’s really just a bookkeeping artifact. In particular, this “NBL context” is *not* tied to the hardware TX descriptor (with the confusingly-similar name xmpTxNBufCtx_t); that resource is allocated later in xmpNicTxServe. The miniport has access to dedicated MiniportReserved fields on the NBL and each NB, in which it ought to be able to stash some bookkeeping until a TX descriptor is available.

The idealized NDIS6 miniport looks like the below. Note that no resources are required to queue the packet, and the only extra bookkeeping buffer is allocated alongside the TX descriptor. It’s my claim that most simple miniports don’t need to allocate any additional resources just to queue the NBL; bookkeeping resources are only needed just before sending.

MiniportSendNetBufferLists(NBL chain) :
for each NBL in chain :
for each NB on NBL :
queue NB [using MiniportReserved for linked list]
PumpQueuedPackets() :
atomically acquire TX descriptor and pop next NB
if no TX descriptor or no NB :
return from PumpQueuedPackets
associate TX descriptor and NB
NdisMAllocateNetBufferSGList :
(ndis does stuff)
MiniportSGListCallback() :
program TX descriptor with memory

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@live.com
Sent: Sunday, October 31, 2010 10:04 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Query about deserialized NDIS miniports

Hey Jeffrey

Many thanks for your quick response :slight_smile:

Umm… Then why does NicTxNbl() return (and complete) a TX NBL with an error in lines 2078 - 2085 ?

XF_Q_DEL_HEAD(&pFifo->NBLCtxQ, link);
if ( !link )
{ /* out of NBL contexts, can’t even queue this NBL. */
XF_DPR_FREE_SLOCK(&pFifo->TxLock);
Status = NDIS_STATUS_RESOURCES;
pFifo->Stats.TxNBLUnder++;
/* XMPTRACE(XMP_DBG_WRN, (“xmpNicTxNBL:Out of NBL Contexts\n”)); */
/* ASSERT(FALSE); */
break;
}

BTW, the actual send complete with error is in the same fn @ lines 2111 - 2115… ‘Serve’ is not set, so xmpNicTxServe() is not called. Is this then a bug in the sample code?


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Wow. Thanks for taking the time, Jeffrey.

Hmm… so ‘never do what the samples do’.

kidding! could not resist that :slight_smile:

On a more serious note:

PumpQueuedPackets() :
atomically acquire TX descriptor and pop next NB
if no TX descriptor or no NB :
return from PumpQueuedPackets

what now…?

when do these these packets go out? what if there’s no RX traffic at all (no ints) nor a ‘TX done’ int either ?

This use case is not all that contrived. A ‘send only’ app (stock quotes) comes to mind.

> ‘never do what the samples do’.

That’s not really the moral here… xframeii is a “real world” driver that supports a gazillion features. One of those features is that its hardware can (in some cases) send multiple packets per TX descriptor, which results in the complicated queuing you see in the driver. We’re showing it all to you, warts and everything. It’s like your opportunity to see what one of the “big guys” does. (It even has an OS abstraction layer, although the kit only includes the Windows bindings…). Of course, for simple drivers, it’s way overkill, which is why I recommend netvmini as a first step in learning. (Another one of netvmini’s advantages: it doesn’t require two US$5000 network adapters before you can start playing with it.)

when do these these packets go out?

Well, in my example, it’s your driver’s job to program a DMA request into your hardware. Usually, you’ll just write some memory addresses into the hardware-visible portion of a TX descriptor, and then poke some register that kicks the hardware into packet-sending action. When the hardware has completed sending the packet (or actually, has completed copying the packet out of main memory and into its own little TX buffer), it will trigger an interrupt. Your MiniportInterrupt (or MiniportMessageInterrupt) handler sees the interrupt, and (using some hardware-specific mechanism) divines that the transmit has completed. Since your hardware probably transmits one NB per TX descriptor, you have to do a little bit of bookkeeping to decide whether that NB was the last NB on the NBL. If so, you complete the NBL back to the OS.

what if there’s no RX traffic at all (no ints) nor a ‘TX done’ int either ?

Well if your hardware really doesn’t have any interrupts for “TX done”, then you get to poll. (Hopefully your hardware at *least* writes transmit status into a register somewhere that you can poll, or you really need to go have a chat with your silicon people!)

This use case is not all that contrived. A ‘send only’ app (stock quotes) comes to mind.

Right – any driver design that depends on RX traffic to keep the TX path moving along is broken. (RX is not a clock source for TX). I’ve debugged cases where something like this happens (by accident) and customers are *not happy* with what appears to be unreproducable and random failures.

By the way, all the discussion here applies only to pure NDIS miniports with physical hardware. IM drivers and NDIS-WDM miniports will do slightly different things at the bottom edge of their send path (e.g., they won’t call NdisMAllocateNetBufferSGList, and they won’t have a MiniportInterrupt handler).

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@live.com
Sent: Sunday, October 31, 2010 10:56 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Query about deserialized NDIS miniports

Wow. Thanks for taking the time, Jeffrey.

Hmm… so ‘never do what the samples do’.

kidding! could not resist that :slight_smile:

On a more serious note:

PumpQueuedPackets() :
atomically acquire TX descriptor and pop next NB
if no TX descriptor or no NB :
return from PumpQueuedPackets

what now…?

when do these these packets go out? what if there’s no RX traffic at all (no ints) nor a ‘TX done’ int either ?

This use case is not all that contrived. A ‘send only’ app (stock quotes) comes to mind.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer