DMA for Read/Write simultaneously

NTDEV
1

How to set up the PCI Bus Master DMA operations so that simultaneous R/W can
be achieved for the WDM driver?

Thanks

Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp.

2

Either use multiple dma adapter objects or use scatter gather lists and
have one active for reads and one for writes.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ta H.
Sent: Monday, April 15, 2002 9:36 PM
To: NT Developers Interest List
Subject: [ntdev] DMA for Read/Write simultaneously

How to set up the PCI Bus Master DMA operations so that
simultaneous R/W can
be achieved for the WDM driver?

Thanks

Get your FREE download of MSN Explorer at
http://explorer.msn.com/intl.asp.

You are currently
subscribed to ntdev as: xxxxx@hollistech.com To unsubscribe
send a blank email to %%email.unsub%%

3

Do you mean by calling IoGetDmaAdapter()twice, so one is for reads and the
other one is for writes. Thus there will be two
DmaOperations->allocateAdapterChannel’s and DmaOperations->MapTransfer’s.
Right? How the Windows take care of the mapping registers if I have two
MapTransfer functions? By the way, does Win98’s WDM fully support SG lists?

Thanks.

Ta H.

From: “Mark Roddy”
>Reply-To: “NT Developers Interest List”
>To: “NT Developers Interest List”
>Subject: [ntdev] RE: DMA for Read/Write simultaneously
>Date: Mon, 15 Apr 2002 23:14:50 -0400
>
>Either use multiple dma adapter objects or use scatter gather lists and
>have one active for reads and one for writes.
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Ta H.
> > Sent: Monday, April 15, 2002 9:36 PM
> > To: NT Developers Interest List
> > Subject: [ntdev] DMA for Read/Write simultaneously
> >
> >
> > How to set up the PCI Bus Master DMA operations so that
> > simultaneous R/W can
> > be achieved for the WDM driver?
> >
> > Thanks
> >
> >
> > Get your FREE download of MSN Explorer at
> > http://explorer.msn.com/intl.asp.
> >
> >
> > —
> > You are currently
> > subscribed to ntdev as: xxxxx@hollistech.com To unsubscribe
> > send a blank email to %%email.unsub%%
> >
>
>
>—
>You are currently subscribed to ntdev as: xxxxx@hotmail.com
>To unsubscribe send a blank email to %%email.unsub%%
Join the world’s largest e-mail service with MSN Hotmail.
http://www.hotmail.com

4
  1. Create an adapter object in MN_START_DEVICE path. IoGetDmaAdapter or query the BUS_INTERFACE_STANDARD interface from the PDO and
    call its method.
  2. Call ->AllocateAdapterChannel in the IO path, specify the number of pages in the MDL there.
  3. In the the AdapterControl callback, call ->MapTransfer in a loop to build a scatter-gather list.
    On w2k and XP, you can use ->GetScatterGatherList instead of steps 2 and 3. It is not supported on Win9x.
  4. Feed the scatter-gather list to the device.
  5. Return DeallocateObjectKeepRegisters from AdapterControl.
  6. When the DMA is done (usually in DpcForIsr), call ->FlushAdapterBuffers (or ->PutScatterGatherList), then ->FreeMapRegisters.

Another technique (used sometimes, NDIS.SYS seems to use it for NIC miniports) is to preallocate the map registers in
MN_START_DEVICE path, just after IoGetDmaAdapter.
Then just call ->MapTransfer, and never call ->FreeMapRegisters till MN_REMOVE_DEVICE.

Max

----- Original Message -----
From: “Ta H.”
To: “NT Developers Interest List”
Sent: Tuesday, April 16, 2002 5:36 AM
Subject: [ntdev] DMA for Read/Write simultaneously

> How to set up the PCI Bus Master DMA operations so that simultaneous R/W can
> be achieved for the WDM driver?
>
> Thanks
>
> _________________________________________________________________
> Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp.
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to %%email.unsub%%
>

5

> MapTransfer functions? By the way, does Win98’s WDM fully support SG lists?

Does not support at all.

Max

6

> Either use multiple dma adapter objects or use scatter gather lists and

Single DMA adapter object (at least on PCI) can sustain any number of concurrent reads/writes.
Just do not set the limit on map registers number in IoGetDmaAdapter.

Max

7

If I have two Queues; one is System provided, and the other is Custom’s.
Thus there could be two CurrentIrps for reads and writes. My question is is
it enough? I mean with one dma adapter, one adaptercontrol routine with one
maptransfer call, can I simultaneously have reads and writes DMA’s? How
about the case that I need to set up staged trnasfers due to the lacking of
the mapping registers?

Thanks

From: “Maxim S. Shatskih”
>Reply-To: “NT Developers Interest List”
>To: “NT Developers Interest List”
>Subject: [ntdev] Re: DMA for Read/Write simultaneously
>Date: Tue, 16 Apr 2002 15:59:54 +0400
>
>1) Create an adapter object in MN_START_DEVICE path. IoGetDmaAdapter or
>query the BUS_INTERFACE_STANDARD interface from the PDO and
>call its method.
>2) Call ->AllocateAdapterChannel in the IO path, specify the number of
>pages in the MDL there.
>3) In the the AdapterControl callback, call ->MapTransfer in a loop to
>build a scatter-gather list.
>On w2k and XP, you can use ->GetScatterGatherList instead of steps 2 and 3.
>It is not supported on Win9x.
>4) Feed the scatter-gather list to the device.
>5) Return DeallocateObjectKeepRegisters from AdapterControl.
>6) When the DMA is done (usually in DpcForIsr), call ->FlushAdapterBuffers
>(or ->PutScatterGatherList), then ->FreeMapRegisters.
>
>Another technique (used sometimes, NDIS.SYS seems to use it for NIC
>miniports) is to preallocate the map registers in
>MN_START_DEVICE path, just after IoGetDmaAdapter.
>Then just call ->MapTransfer, and never call ->FreeMapRegisters till
>MN_REMOVE_DEVICE.
>
> Max
>
>----- Original Message -----
>From: “Ta H.”
>To: “NT Developers Interest List”
>Sent: Tuesday, April 16, 2002 5:36 AM
>Subject: [ntdev] DMA for Read/Write simultaneously
>
>
> > How to set up the PCI Bus Master DMA operations so that simultaneous R/W
>can
> > be achieved for the WDM driver?
> >
> > Thanks
> >
> >
> > Get your FREE download of MSN Explorer at
>http://explorer.msn.com/intl.asp.
> >
> >
> > —
> > You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> > To unsubscribe send a blank email to %%email.unsub%%
> >
>
>
>—
>You are currently subscribed to ntdev as: xxxxx@hotmail.com
>To unsubscribe send a blank email to %%email.unsub%%
Chat with friends online, try MSN Messenger: http://messenger.msn.com

8

> it enough? I mean with one dma adapter, one adaptercontrol routine with one

maptransfer call, can I simultaneously have reads and writes DMA’s?

Surely.

How
about the case that I need to set up staged trnasfers due to the lacking of
the mapping registers?

I don’t think you will meet this on PCI/x86, but in this case AdapterControl will just be delayed.

Max

9

“Ta H.” wrote in message news:xxxxx@ntdev…
>
> If I have two Queues; one is System provided, and the other is Custom’s.
> Thus there could be two CurrentIrps for reads and writes. My question is
is
> it enough? I mean with one dma adapter, one adaptercontrol routine with
one
> maptransfer call, can I simultaneously have reads and writes DMA’s? How
> about the case that I need to set up staged trnasfers due to the lacking
of
> the mapping registers?
>

It’s really not QUITE as simple in all cases as people are making it out.

The number of map registers you get back from IoGetDmaAdapter(…) is the
maximum number of map registers you’re allowed to use for ONE transfer. So,
you can allocate either one DMA adapter for read and one for write, or you
can just use one DMA adapter. It really doesn’t make too much difference.
One might argue that using two adapters helps “hint” the hal that you intend
to process multiple transfers simultaneously, but… it doesn’t really
matter. So THAT’s not a problem.

There is a well known issue with AllocateAdapterChannel; If a transfer must
pend due to lack of “map registers” (that is, the callback to your
ExecutionRoutine is delayed) only one transfer at a time may pend per device
object. This is because there’s only one WAIT_CONTEXT_BLOCK structure in
your device object, and AllocateAdapterChannel unconditionally uses it to
pend your request. An old trick here is using “dummy” device objects to
avoid this problem… Yuck! GetScatterGatherList does not do this (it
allocates its WAIT_CONTEXT_BLOCK from pool), and thus avoids this problem.
This is a major advantage of GetScatterGatherList.

And Max: The way I understand it, you CAN see simultaneous read and write
DMA operations on PCI/X86 from the same device. Understand, you can’t see
them LITERALLY simultaneously. But, as far as I understand it, the bursts
can be interleaved (or something – somebody who’s a real hardware engineer
wanna give us the definitive answer and straighten us software fools out on
this, please).

So… in answer to your question Ta H.: Just fire 'em off simultaneously.
Just be SURE not to pend more than one.

Peter
OSR

10

>From: “Peter Viscarola”
>Reply-To: “NT Developers Interest List”
>To: “NT Developers Interest List”
>Subject: [ntdev] Re: DMA for Read/Write simultaneously
>Date: Tue, 16 Apr 2002 17:32:56 -0400
>
>
>It’s really not QUITE as simple in all cases as people are making it out.
>
>The number of map registers you get back from IoGetDmaAdapter(…) is the
>maximum number of map registers you’re allowed to use for ONE transfer.
>So,
>you can allocate either one DMA adapter for read and one for write, or you
>can just use one DMA adapter. It really doesn’t make too much difference.
>One might argue that using two adapters helps “hint” the hal that you
>intend
>to process multiple transfers simultaneously, but… it doesn’t really
>matter. So THAT’s not a problem.
>
>There is a well known issue with AllocateAdapterChannel; If a transfer
>must
>pend due to lack of “map registers” (that is, the callback to your
>ExecutionRoutine is delayed) only one transfer at a time may pend per
>device
>object. This is because there’s only one WAIT_CONTEXT_BLOCK structure in
>your device object, and AllocateAdapterChannel unconditionally uses it to
>pend your request. An old trick here is using “dummy” device objects to
>avoid this problem… Yuck! GetScatterGatherList does not do this (it
>allocates its WAIT_CONTEXT_BLOCK from pool), and thus avoids this problem.
>This is a major advantage of GetScatterGatherList.
>
Since the driver has to support Win98, 2k, and XP, SGList cannot be my
choice. Peter, would you mind elaborating on using “dummy” device objects
for the case?

Thanks

Ta. H

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp.

11

“Ta H.” wrote in message news:xxxxx@ntdev…
>
> choice. Peter, would you mind elaborating on using “dummy” device objects
> for the case?
>

Call IoCreateDevice(…) to make additional device objects. You leave these
unnamed. I haven’t done this in ages (read: Since GetScatterGatherList was
supported) so I can’t say for sure if you have to attach
(IoAttachDeviceToDeviceStack) them to a PDO or not… I wouldn’t think you’d
have to (just try it to see).

You can narrow your problem down by using (for example) your REAL Device
Object for your reads and your dummy Device Object for your writes. Make
sure you write your code so that only one read and one write request can be
in progress on your adapter at a time – In other words, pend the IRPs
awaiting a free transfer slot.

Do that, you should be golden,

Peter
OSR

12

Thanks. So my understanding is I can create an unnamed device object for
writes (for example) by IoCreateDevice. Then I use that device object to
obtain an extra DMA adapter for writes. Last I will need to have an extra
AdapterControl routine for writes where there is one MapTransfer. Right? By
the way, what should I return, DeallocateObjectKeepRegsters or KeepObject?
(from reads’ and writes’AdapterControl routines)

Thanks

Ta H.

From: “Peter Viscarola”
>Reply-To: “NT Developers Interest List”
>To: “NT Developers Interest List”
>Subject: [ntdev] Re: DMA for Read/Write simultaneously
>Date: Tue, 16 Apr 2002 18:00:00 -0400
>
>“Ta H.” wrote in message news:xxxxx@ntdev…
> >
> > choice. Peter, would you mind elaborating on using “dummy” device
>objects
> > for the case?
> >
>
>Call IoCreateDevice(…) to make additional device objects. You leave
>these
>unnamed. I haven’t done this in ages (read: Since GetScatterGatherList
>was
>supported) so I can’t say for sure if you have to attach
>(IoAttachDeviceToDeviceStack) them to a PDO or not… I wouldn’t think
>you’d
>have to (just try it to see).
>
>You can narrow your problem down by using (for example) your REAL Device
>Object for your reads and your dummy Device Object for your writes. Make
>sure you write your code so that only one read and one write request can be
>in progress on your adapter at a time – In other words, pend the IRPs
>awaiting a free transfer slot.
>
>Do that, you should be golden,
>
>Peter
>OSR
>
>
>
>
>
>
>—
>You are currently subscribed to ntdev as: xxxxx@hotmail.com
>To unsubscribe send a blank email to %%email.unsub%%

_________________________________________________________________
Send and receive Hotmail on your mobile device: http://mobile.msn.com

13

OSR’s NT Insider had an article called Extreme DMA in it about 18 months
ago. It provides a template for 2000 and using GetScatterGatherList. However
it can also be used in NT, and most likely with a little platform specific
coding across the platforms you mentioned.

My original design for our fibre channel boards used 2 separate device
objects, send and receive, each with it’s own DMA adapter object. Having
read OSR’s article I went back to a singe device object and DMA adapter
object. I now allocate an ungodly number of map register, and use a queue
that feeds the DMA adapter object callback function. When a transfer request
is received it is inserted at the bottom of the list and the adapter object
tested for busy. If it is busy, I exit. If not busy than I mark it busy and
allocate the DMA adapter object.

When the callback function runs, it pops the head of the list, uses
IoMapTransfer to create a Scatter/Gather list, fills in a fibre channel
request block (FCRB) and hangs the FCRB on the hardware. The callback runs
until the list is empty and then releases the adapter object with
DeallocateObjectKeepRegisters and resets the busy flag. The list may
comprise a mix of transfer requests that vary in length , directions
(send/receive), and protocols (initiator/target). Our fibre channel adapter
permits up to 64 S/G lists to be processed, simultaneously.

–
Gary G. Little
xxxxx@broadstor.com
xxxxx@inland.net

“Ta H.” wrote in message news:xxxxx@ntdev…
>
> Thanks. So my understanding is I can create an unnamed device object for
> writes (for example) by IoCreateDevice. Then I use that device object to
> obtain an extra DMA adapter for writes. Last I will need to have an extra
> AdapterControl routine for writes where there is one MapTransfer. Right?
By
> the way, what should I return, DeallocateObjectKeepRegsters or KeepObject?
> (from reads’ and writes’AdapterControl routines)
>
> Thanks
>
> Ta H.
>
>
> >From: “Peter Viscarola”
> >Reply-To: “NT Developers Interest List”
> >To: “NT Developers Interest List”
> >Subject: [ntdev] Re: DMA for Read/Write simultaneously
> >Date: Tue, 16 Apr 2002 18:00:00 -0400
> >
> >“Ta H.” wrote in message news:xxxxx@ntdev…
> > >
> > > choice. Peter, would you mind elaborating on using “dummy” device
> >objects
> > > for the case?
> > >
> >
> >Call IoCreateDevice(…) to make additional device objects. You leave
> >these
> >unnamed. I haven’t done this in ages (read: Since GetScatterGatherList
> >was
> >supported) so I can’t say for sure if you have to attach
> >(IoAttachDeviceToDeviceStack) them to a PDO or not… I wouldn’t think
> >you’d
> >have to (just try it to see).
> >
> >You can narrow your problem down by using (for example) your REAL Device
> >Object for your reads and your dummy Device Object for your writes. Make
> >sure you write your code so that only one read and one write request can
be
> >in progress on your adapter at a time – In other words, pend the IRPs
> >awaiting a free transfer slot.
> >
> >Do that, you should be golden,
> >
> >Peter
> >OSR
> >
> >
> >
> >
> >
> >
> >—
> >You are currently subscribed to ntdev as: xxxxx@hotmail.com
> >To unsubscribe send a blank email to %%email.unsub%%
>
>
>
>
> _________________________________________________________________
> Send and receive Hotmail on your mobile device: http://mobile.msn.com
>
>
>

14

“Ta H.” wrote in message news:xxxxx@ntdev…
>
> How to set up the PCI Bus Master DMA operations so that simultaneous R/W
can
> be achieved for the WDM driver?
>

We’ve answered this question already, Ta.

P

15

“Ta H.” wrote in message news:xxxxx@ntdev…
>
> How to set up the PCI Bus Master DMA operations so that simultaneous R/W
can
> be achieved for the WDM driver?
>

Sorry, Ta! My news reader is playing tricks on me…

Apologies… I seem to be getting old messages. Old saying “never change
laptops…”

Peter
OSR