Dma-able memory

NTDEV
1

Hi,
I am currently in the design phase of a project in which we have a dedicated x64 machine that have one or more HBAs plugged in. Those HBAs are all 64 bits capable and support scatter-gather. The HBA ports act as an initiator and target devices at the same time. Target I/O is always received into a big pre-allocated non-paged kernel memory (Can be 0.5gb on x64 machines). Initiator I/O can be sent from anywhere in the machine memory (as the I/O comes from a user-mode daemon). I am now checking the DMA ddi and see how I can put them to the best use. Obviously, for the initiator I/Os side, I can use the standard way of building SG list, I can even fail I/O on resource failure so no problem here. The problem arises for the target I/O side. Since I/O can can arrive from different HBA ports, I cannot use common buffer technique. On the other hand, to start building SG list for each I/O(Even if I use BuildScatterGatherList instead of CreateScatterGatherList) can be a big performance hit. Obviously, the easiest way is to use MmGetPhysicalAddress to get the addresses needed for DMA transactions, however I know that doing this is like sitting on a bomb with a short fuse. Therefore I was thinking about calling BuildScatterGatherList for the sam buffer separately for each of the HBA ports, and never call PutScatterGatherList.
My questions are:
1.Is it possible to call BuildScatterGatherList more than once on the same non-paged memory portion for different ports?
2.Since I don’t want to call PutScatterGatherList, I still need to somehow do all the necessary flushing operations . In this case, can I use FlushAdapterBuffers instead?
3.I plan to use KWDF for the driver, however I believe I will have to revert to WDM at least to deal with the DMA operations since KWDF expects WDFREQUEST, which I won’t allocate for the target I/Os. Is there some way to use KWDF for DMA without WDFREQUEST?

Thanks,
Eran.

2

You call PutScatterGatherList which knows that you allocated the SGL because
you called BuildScatterGatherList.

I see no reason why you cannot have several SGLs each created with
BuildScatterGatherList all describing overlapping regions of memory.

See the PLxEvtIoWrite sample routine for clues on how to use KMDF to do DMA
without having a WdfRequest object. WdfDmaTransactionInitialize uses an mdl
not a wdfrequest.

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@topio.com
Sent: Wednesday, September 27, 2006 5:11 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Dma-able memory

Hi,
I am currently in the design phase of a project in which we
have a dedicated x64 machine that have one or more HBAs
plugged in. Those HBAs are all 64 bits capable and support
scatter-gather. The HBA ports act as an initiator and target
devices at the same time. Target I/O is always received into
a big pre-allocated non-paged kernel memory (Can be 0.5gb on
x64 machines). Initiator I/O can be sent from anywhere in the
machine memory (as the I/O comes from a user-mode daemon). I
am now checking the DMA ddi and see how I can put them to the
best use. Obviously, for the initiator I/Os side, I can use
the standard way of building SG list, I can even fail I/O on
resource failure so no problem here. The problem arises for
the target I/O side. Since I/O can can arrive from different
HBA ports, I cannot use common buffer technique. On the other
hand, to start building SG list for each I/O(Even if I use
BuildScatterGatherList instead of CreateScatterGatherList)
can be a big performance hit. Obviously, the easiest way is
to use MmGetPhysicalAddress to get the addresses needed for
DMA transactions, however I know that doing this is like
sitting on a bomb with a short fuse. Therefore I was thinking
about calling BuildScatterGatherList for the sam buffer
separately for each of the HBA ports, and never call
PutScatterGatherList.
My questions are:
1.Is it possible to call BuildScatterGatherList more than
once on the same non-paged memory portion for different ports?
2.Since I don’t want to call PutScatterGatherList, I still
need to somehow do all the necessary flushing operations . In
this case, can I use FlushAdapterBuffers instead?
3.I plan to use KWDF for the driver, however I believe I will
have to revert to WDM at least to deal with the DMA
operations since KWDF expects WDFREQUEST, which I won’t
allocate for the target I/Os. Is there some way to use KWDF
for DMA without WDFREQUEST?

Thanks,
Eran.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online
at http://www.osronline.com/page.cfm?name=ListServer

3

Thanks for the reply.
The issue with calling PutScatterGatherList is that it frees all the map registers allocated for the transfer, which is something I don’t want to happen, and therefore I was searching for a way to still do all the flushing without releasing the mapping.
So is it legit to call FlushAdapterBuffers(Or whatever KWDF equivalet if there is one) after each DMA transfer is done, and only when the device is deleted, to call PutScatterGatherList(or WdfObjectDelete or the transaction object).

Thanks,
Eran.

4

It is only going to free ‘map registers’ if it allocated real ‘map
registers’ and if it did allocate real ‘map registers’ those are scarce
resources that you cannot keep permanently for your use alone.

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@topio.com
Sent: Wednesday, September 27, 2006 11:03 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Dma-able memory

Thanks for the reply.
The issue with calling PutScatterGatherList is that it frees
all the map registers allocated for the transfer, which is
something I don’t want to happen, and therefore I was
searching for a way to still do all the flushing without
releasing the mapping.
So is it legit to call FlushAdapterBuffers(Or whatever KWDF
equivalet if there is one) after each DMA transfer is done,
and only when the device is deleted, to call
PutScatterGatherList(or WdfObjectDelete or the transaction object).

Thanks,
Eran.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online
at http://www.osronline.com/page.cfm?name=ListServer