limitations on DMA transfer size?

NTDEV
1

I am writing a packet DMA driver on Win7-64 OS for a 64-bit PCIx data acquisition device. I am having a problem with large DMAs. I am hitting a ceiling of about 2GB max transfer size. Our max size req is 5GB for a single DMA transfer. Is this possible? Has anyone achieved such a transfer size?

I have plenty of RAM (24GB). I can alloc a 2.3GB receive buffer (and for my testing I am doing send as well, so I am allocing a 2.3GB send buffer as well). It appears that I can call GetScatterGatherList successfully for my full transfer upfront. I am using a descriptor ring with only 200 descriptors which works nicely under 2GB.

I get some data errors just above 2GB (all 0’s). And then no data written to the rest of the buffer.

I’d be happy to provide more info, but I am really curious to the question: has anyone ever done DMAs this large?

Thanks,
Jeff

2

Hi Jeff,

For completeness, I’ll ask (though I suspect I know the answer): Does your device support scatter/gather in hardware? Can you show us how you’re initializing your DEVICE_DESCRIPTION data structure?

What’s the maximum number of map registers that’s returned to you from IoGetDmaAdapter?

I’m also not sure from your note of what the problem is that you’re experiencing, exactly. You said “I get some data errors just above 2GB (all 0’s). And then no data written to the rest of the buffer.” – Is THAT the problem you’re having?

Also, I have to ask: You’re quite confident that the actual DMA device is programmed correctly and is working correctly?

Peter
OSR

3

>I get some data errors just above 2GB (all 0’s). And then no data written to the rest of the buffer.
What kind I/O is it? Read or Write?
You also need to check if your hardware work correctly. Try to use a PCI bus analyzer and any debug tools which you could use on your hardware.

Igor Sharovar

4

>Hi Jeff,

For completeness, I’ll ask (though I suspect I know the answer): Does your
device support scatter/gather in hardware? Can you show us how you’re
initializing your DEVICE_DESCRIPTION data structure?

What’s the maximum number of map registers that’s returned to you from
IoGetDmaAdapter?

Yes, the device supports HW scatter/gather. I am not using a DEVICE_DESCRIPTION structure. That is, I am not calling IoGetDmaAdapter(). This is a WDF driver so I am calling WdfDmaEnablerCreate() followed by WdfDmaEnablerGetFragmentLength(). The fragment length returned is 0x40000000 (1GB). (hmmm…) That translates to 0x40000 pages (thus 0x40000 map registers ?).

Before calling WdfDmaEnablerCreate(), I call:

WDF_DMA_ENABLER_CONFIG_INIT(&dmaConfig,
WdfDmaProfileScatterGather64,
pDmaDevExt->MaximumTransferLength);

where pDmaDevExt->MaximumTransferLength is set to 0x140000000 (5GB)

I’m also not sure from your note of what the problem is that you’re
experiencing, exactly. You said “I get some data errors just above 2GB (all
0’s). And then no data written to the rest of the buffer.” – Is THAT the
problem you’re having?

That is the application observable problem. In the driver, I appear to stop getting interrupts at some point from the receive DMA interrupt source (set to interrupt on EOP (end of packet)). Not sure what else is happening/not happening…

Also, I have to ask: You’re quite confident that the actual DMA device is
programmed correctly and is working correctly?

Actually, I am not sure if the device is working correctly. We are using a 3rd party FPGA IP core for the DMA engine and the engineers have indicated they have not tried transfers this large.

I am confident the device and my control of it is working properly for transfer sizes of 2GB and less. Actually, sizes of 0x80017DFC bytes are working consistently. I have a simple 4 byte incrementing data pattern comparison test that succeeds.

Peter
OSR

5

>What kind I/O is it? Read or Write?

You also need to check if your hardware work correctly. Try to use a PCI bus
analyzer and any debug tools which you could use on your hardware.

Igor Sharovar

Our application is for Receive (Card-to-System Memory). But for testing we have a Send channel that loops back into the receive. So I am doing both Send and Receive. However, on the send side I am doing several single-packet transfers. And on the Rcv side, just one honking transfer of all packets. My packet size is 58436 bytes.

Unfortunately I don’t have a PCIx bus analyzer. Though I may be able to get the HW engineer to set up chip scope for exposure into the device FPGA.

~Jeff

6

Ah! You’re using KMDF. Good idea.

Once again, I want to be sure I’m following what you’re saying (and answering the right question!). You said you called WdfDmaEnablerGetFragmentLength and it returned “0x40000000 (1GB)” – So, the OS is telling you that the maximum transfer size you can create is 1GB.

Are you trying to transfer >2GB, after being told you can only transfer 1GB??

Peter
OSR

7

And there is nothing wrong with building multiple partial SGLs out of your giant MDL. I believe. Though I’m not quite sure why you want such a big single transfer. Are you concerned about buffer underrun?

8

Absolutely correct.

That’s what Windows expects you to do. If WdfDmaEnablerGetFragmentLength returns 1GB, and you have 5GB that need transferred, the OS is telling you to do the DMA operation in chunks not exceeding 1GB at a time. It’s just part of the logic you need to include in your driver.

Peter
OSR

9

>WdfDmaEnablerGetFragmentLength(). The fragment length returned is 0x40000000 (1GB). >(hmmm…) That translates to 0x40000 pages (thus 0x40000 map registers ?).
Have you checked max number of available map register? I mean, using the macro BYTE_TO_PAGES(WdfDmaEnablerGetFragmentLength()) + 1.
Look again the documentation of WdfDmaEnablerGetFragmentLength. You likely got max available memory.

Igor Sharovar

10

Argg, my last response got lost due to auto-logout… :stuck_out_tongue:

Yes, Peter, you are correct. I am attempting to transfer more than it tells me. Since I was able to transfer 2GB I was assuming (ha!) that the 1GB limit it told me didn’t really matter.

Upon further investigation, I have confirmed that my call to WDF_DMA_ENABLER_CONFIG_INIT() to set the MaxLength is only taking in a 4 byte param for MaxLength, not the 8 byte “size_t” as I expected. E.g. when I specify 4GB, I get back 0 bytes from WdfDmaEnablerGetMaximumLength(). When I specify 5GB, I get back 1GB… Could this be a bug? I can’t see what I am doing wrong here.

So, when I specify a MaxLength of 0xfffff000 (4GB - PAGE_SIZE), WdfDmaEnablerGetFragmentLength() gives me back 0x7ffff00 for 0x80000 map registers. This makes more sense as to the 2GB limit I am seeing.

Do I have any control over this? How can I make it bigger? Is this limit coming from OS or board?

Thanks-
Jeff

11

Right now, our data source has no flow control (hardware handshake), so, yes I am worried about not keeping up and losing data. If the OS has a hard limit of how much can be transferred in one DMA, then, I will have to take it back to the hardware guys and break up the transfer into chunks.

Sorry, I am not quite sure I follow “building multiple partial SGLs out of your giant MDL”. Right now, for my Packet DMA with 58436 byte packets, I am calling GetScatterGatherList once for each packet. When I tried to put PutScatterGatherList/GetScatterGatherList in my DPC, I kept getting BugCheck on GetScatterGatherList. I figured that it was because I was in the wrong context from the buffer alloc context. (buffer virt addresses were wrong)

So now I call GetScatterGatherList on the whole thing up front. And Put… when all done.

12

Sorry, WdfDmaEnablerGetFragmentLength() gives me back 0x7ffff000 (2GB - PAGE_SIZE) (not 0x7ffff00)…

13

Jeff,

The OS is giving you a limit when you call WdfDmaEnablerGetFragmentLength – that’s the point of making this call. Even if NTOS *let* you “get away” with a larger transfer, ignoring the limit is a serious architectural error in your driver and driver verifier (you ARE running with Driver Verifier enabled on your test machine, right?? And you’re running it with everything enabled, including DMA verifier, WITH THE EXCEPTION OF Low Resource Simulation???) will hammer you for the violation. You should have both KMDF Verifier and Windows Driver Verifier enabled AT ALL TIME when you’re developing and testing… there’s really no downside to doing this, and LOTS of mistakes will get caught early. OK, I’ll get off my soap-box now. I’m really writing for the archives here, in any case).

In a word, no.

Ah, the question sooo many people ask. The answer is, as far as I know you can’t.

It’s coming from the OS… though, given that you have a 64-bit, device that does scatter/gather, I’m a bit surprised you’re seeing this limit.

Peter
OSR

14

On 11/04/2010 03:15 AM, xxxxx@osr.com wrote:

In a word, no.

Ah, the question sooo many people ask. The answer is, as far as I know you can’t.

Of course he can:
(1) Buy Microsoft
(2) Order to have this limit changed
(and the change rolled out as a hotfix)

Probably impracticable, but *theoretically* possible. :wink:

15

Thank you for the help - though I don’t think I will be buying microsoft right yet :wink:

One reason we went to 64 bit platform was to defeat these memory limitations. Win7-64 seems to be giving me the same limitation that XP has…

I am not giving up yet. :wink: There must be something I can look at. Any suggestions???

Peter, your statement

“given that you have a 64-bit, device that does scatter/gather, I’m a bit surprised you’re seeing this limit”

leads me to believe that others have defeated this - now I need to find out what is different b/t theirs and mine…

Is there any clue from the fact that I specify an 8 byte param (size_t) to WDF_DMA_ENABLER_CONFIG_INIT(), but when I ask for that param back via WdfDmaEnablerGetMaximumLength(), it gives me the param back with the top 4 bytes chopped off… I mean, my understanding is that WdfDmaEnablerGetMaximumLength() gives me what I told it, and WdfDmaEnablerGetFragmentLength() gives me the Windows adjusted value.

(I am building in Win7 x64 Checked Build Environment…)

16

You might have answered this already but is your device on 32 or 64 bit bus. If it is on a 32 bit bus, does it do 64 bit addressing?

–Mark Cariddi
OSR Open Systems Resources, Inc.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@neurologica.com
Sent: Thursday, November 04, 2010 11:42 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] limitations on DMA transfer size?

Thank you for the help - though I don’t think I will be buying microsoft right yet :wink:

One reason we went to 64 bit platform was to defeat these memory limitations. Win7-64 seems to be giving me the same limitation that XP has…

I am not giving up yet. :wink: There must be something I can look at. Any suggestions???

Peter, your statement

“given that you have a 64-bit, device that does scatter/gather, I’m a bit surprised you’re seeing this limit”

leads me to believe that others have defeated this - now I need to find out what is different b/t theirs and mine…

Is there any clue from the fact that I specify an 8 byte param (size_t) to WDF_DMA_ENABLER_CONFIG_INIT(), but when I ask for that param back via WdfDmaEnablerGetMaximumLength(), it gives me the param back with the top 4 bytes chopped off… I mean, my understanding is that WdfDmaEnablerGetMaximumLength() gives me what I told it, and WdfDmaEnablerGetFragmentLength() gives me the Windows adjusted value.

(I am building in Win7 x64 Checked Build Environment…)

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

17

Let’s back up a bit here.

You said:

It’s kinda difficult to believe this is a bug in KMDF… You HAVE the sources to the macros that you reference. Check out the definition of WDF_DMA_ENABLER_CONFIG and what WDF_DMA_ENABLER_CONFIG_INIT does:


typedef struct _WDF_DMA_ENABLER_CONFIG {
//
// Size of this structure in bytes
//
ULONG Size;

//
// One of the above WDF_DMA_PROFILES
//
WDF_DMA_PROFILE Profile;

//
// Maximum DMA Transfer handled in bytes.
//
size_t MaximumLength;

//
// The various DMA PnP/Power event callbacks
//
PFN_WDF_DMA_ENABLER_FILL EvtDmaEnablerFill;
PFN_WDF_DMA_ENABLER_FLUSH EvtDmaEnablerFlush;
PFN_WDF_DMA_ENABLER_DISABLE EvtDmaEnablerDisable;
PFN_WDF_DMA_ENABLER_ENABLE EvtDmaEnablerEnable;
PFN_WDF_DMA_ENABLER_SELFMANAGED_IO_START EvtDmaEnablerSelfManagedIoStart;
PFN_WDF_DMA_ENABLER_SELFMANAGED_IO_STOP EvtDmaEnablerSelfManagedIoStop;

} WDF_DMA_ENABLER_CONFIG, *PWDF_DMA_ENABLER_CONFIG;

VOID
FORCEINLINE
WDF_DMA_ENABLER_CONFIG_INIT(
out PWDF_DMA_ENABLER_CONFIG Config,
 in WDF_DMA_PROFILE Profile,
__in size_t MaximumLength
)
{
RtlZeroMemory(Config, sizeof(WDF_DMA_ENABLER_CONFIG));

Config->Size = sizeof(WDF_DMA_ENABLER_CONFIG);
Config->Profile = Profile;
Config->MaximumLength = MaximumLength;
}


Kinda hard for me to believe that there’s a bug there.

And I can confirm the WdfDmaEnablerGetMaximumLength just returns the value from the MaximumLength field of the WDF_DMA_ENABLER_CONFIG object.

Soooo… sounds to me like you’ve got an issue with casting or whatever?

Peter
OSR

18

Did you using the WDM dma adapter to see of kmdf is giving you problems?

d

dent from a phpne with no keynoard

-----Original Message-----
From: xxxxx@neurologica.com
Sent: November 04, 2010 8:41 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] limitations on DMA transfer size?

Thank you for the help - though I don’t think I will be buying microsoft right yet :wink:

One reason we went to 64 bit platform was to defeat these memory limitations. Win7-64 seems to be giving me the same limitation that XP has…

I am not giving up yet. :wink: There must be something I can look at. Any suggestions???

Peter, your statement

“given that you have a 64-bit, device that does scatter/gather, I’m a bit surprised you’re seeing this limit”

leads me to believe that others have defeated this - now I need to find out what is different b/t theirs and mine…

Is there any clue from the fact that I specify an 8 byte param (size_t) to WDF_DMA_ENABLER_CONFIG_INIT(), but when I ask for that param back via WdfDmaEnablerGetMaximumLength(), it gives me the param back with the top 4 bytes chopped off… I mean, my understanding is that WdfDmaEnablerGetMaximumLength() gives me what I told it, and WdfDmaEnablerGetFragmentLength() gives me the Windows adjusted value.

(I am building in Win7 x64 Checked Build Environment…)

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

19

The following code:

WDF_DMA_ENABLER_CONFIG_INIT(&dmaConfig,
WdfDmaProfileScatterGather64,
pDmaDevExt->MaximumTransferLength); // set to 0x140000000 or 5GB

status = WdfDmaEnablerCreate(pDevExt->Device,
&dmaConfig,
WDF_NO_OBJECT_ATTRIBUTES,
&pDmaDevExt->DmaEnabler);

if (NT_SUCCESS(status))
{
size_t maxLen = WdfDmaEnablerGetMaximumLength(pDmaDevExt->DmaEnabler);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“dmaConfig.MaximumLength is %lld (0x%x%08x)”, dmaConfig.MaximumLength,
dmaConfig.MaximumLength >> 32,
dmaConfig.MaximumLength);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“WdfDmaEnablerGetMaximumLength returns %lld (0x%x%08x)”,
maxLen, maxLen >> 32, maxLen);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“pDmaDevExt->MaximumTransferLength is %lld (0x%x%08x) (sizeof(size_t) is %d)”,
pDmaDevExt->MaximumTransferLength,
pDmaDevExt->MaximumTransferLength >> 32,
pDmaDevExt->MaximumTransferLength, sizeof(size_t));

… give me the following log entries…

dmaConfig.MaximumLength is 5368709120 (0x140000000)
WdfDmaEnablerGetMaximumLength returns 1073741824 (0x040000000)
pDmaDevExt->MaximumTransferLength is 5368709120 (0x140000000) (sizeof(size_t) is 8)

The only conclusion that I can see from this is that the enabler only took in 4 bytes of the size_t param…

20

>Did you using the WDM dma adapter to see of kmdf is giving you problems?

I will try this, thanks.