My question is - what would be the proper way to implement the same “zerocopy” approach on Rx path? At the moment it works like this:
packet is received by WskReceiveFromEvent callback
packet is decrypted into ciphertext buffer, fetched from pre-allocated “producer” pool (I use DMF_BufferQueue)
ciphertext buffer is enqueued into “consumer” queue
call NetRxQueueNotifyMoreReceivedPacketsAvailable() to trigger Rx queue’s Advance callback
in Rx Advance callback, iterate over fragments, dequeue buffer from “consumer” queue and copy buffer content to fragment’s VA
buffer is “reused” by placing into “producer” pool
I can do decryption in-place and make WSK retain data by returning STATUS_PENDING from WskReceiveFromEvent callback, but how do I “indicate” data provided by WSK to NetAdapter without memcpying? Can I somehow tell NET_FRAGMENT “hey use this MDL which I got from WSK and decrypted in-place” ?
Yeah thanks, that was the case indeed. I also figured it out by reading NetAdapter’s code. Here is how I made it work in my driver:
When setting datapath capabilities, we use NET_ADAPTER_RX_CAPABILITIES_INIT_DRIVER_MANAGED macro. We also specify EvtAdapterReturnRxBuffer callback.
In WskReceiveFromEvent callback we iterate over WSK_DATAGRAM_INDICATION list and enqueue each item into cosumer pool (from DMF_BufferQueue)
// each datagram indication is one UDP datagram
for (PWSK_DATAGRAM_INDICATION next; dataIndication != NULL; dataIndication = next) {
next = dataIndication->Next;
// break list so that we can pass individual WSK_DATAGRAM_INDICATION to WskRelease
dataIndication->Next = NULL;
// fetch buffer from producer
OVPN_RX_BUFFER* rxBuffer;
LOG_IF_NOT_NT_SUCCESS(OvpnBufferQueueFetch(device->DataRxBufferQueue, &rxBuffer));
rxBuffer->DatagramIndication = dataIndication;
// enqueue buffer to consumer, which is consumed by NetAdapter's RX Advance callback
OvpnBufferQueueEnqueue(device->DataRxBufferQueue, rxBuffer);
}
// tell NetAdapter that we have something to consume
OvpnAdapterNotifyRx(device->Adapter);
In RX queue’s Advance callback we iterate over queue’s fragments. We dequeue datagram indication and set fragment properties based on datagram indication’s buffer and MDL:
while (NetFragmentIteratorHasAny(&fi)) {
OVPN_RX_BUFFER* buffer;
// nothing has arrived and decrypted yet?
if (!NT_SUCCESS(OvpnBufferQueueDequeue(bufferQueue, &buffer))) {
break;
}
fragment = NetFragmentIteratorGetFragment(&fi);
PWSK_DATAGRAM_INDICATION datagramIndication = buffer->DatagramIndication;
// return buffer to producer queue
OvpnBufferQueueReuse(bufferQueue, buffer);
PMDL mdl = datagramIndication->Buffer.Mdl;
// TODO: correctly adjust crypto/ovpn overhead
fragment->ValidLength = datagramIndication->Buffer.Length - 8;
fragment->Offset = datagramIndication->Buffer.Offset + 8;
fragment->Capacity = MmGetMdlByteCount(mdl);
NET_FRAGMENT_VIRTUAL_ADDRESS* virtualAddr = NetExtensionGetFragmentVirtualAddress(&queue->VirtualAddressExtension, NetFragmentIteratorGetIndex(&fi));
virtualAddr->VirtualAddress = (PUCHAR)(MmGetSystemAddressForMdlSafe(mdl, LowPagePriority));
// TODO: handle case when packet (DataIndication) contains multiple fragments
NET_PACKET* packet = NetPacketIteratorGetPacket(&pi);
packet->FragmentIndex = NetFragmentIteratorGetIndex(&fi);
packet->FragmentCount = 1;
packet->Layout = {};
// NetAdapter will call ReturnRxBuffer callback when it is done with buffers, there we return datagramIndication back to WSK
returnCtx = NetExtensionGetFragmentReturnContext(&queue->ReturnContextExtension, NetFragmentIteratorGetIndex(&fi));
returnCtx->Handle = (NET_FRAGMENT_RETURN_CONTEXT_HANDLE)datagramIndication;
NetFragmentIteratorAdvance(&fi);
NetPacketIteratorAdvance(&pi);
}
NetFragmentIteratorSet(&fi);
NetPacketIteratorSet(&pi);
In NetAdapter’s EvtAdapterReturnRxBuffer callback we call WskRelease on a datagram indication, which is passed as NET_FRAGMENT_RETURN_CONTEXT_HANDLE by NetAdapter:
NetAdapterCx has some double dozen DV checks. You can enable verifier once and check all o.k. as well.
One thing I hope NetAdapterCx does is burst returning of return conetxts. Right now it returns 1 by 1 thru EvtAdapterReturnRxBuffer. Maybe it can use a ring abstraction (if it doesn’t want to bloatNET_FRAGMENT_RETURN_CONTEXT_HANDLE abstraction) here as well, to burst return bunch of return-contexts insetad of making EvtAdapterReturnRxBuffer callback into driver for every rx-completion.