limitations on DMA transfer size?

Argg, my last response got lost due to auto-logout… :stuck_out_tongue:

Yes, Peter, you are correct. I am attempting to transfer more than it tells me. Since I was able to transfer 2GB I was assuming (ha!) that the 1GB limit it told me didn’t really matter.

Upon further investigation, I have confirmed that my call to WDF_DMA_ENABLER_CONFIG_INIT() to set the MaxLength is only taking in a 4 byte param for MaxLength, not the 8 byte “size_t” as I expected. E.g. when I specify 4GB, I get back 0 bytes from WdfDmaEnablerGetMaximumLength(). When I specify 5GB, I get back 1GB… Could this be a bug? I can’t see what I am doing wrong here.

So, when I specify a MaxLength of 0xfffff000 (4GB - PAGE_SIZE), WdfDmaEnablerGetFragmentLength() gives me back 0x7ffff00 for 0x80000 map registers. This makes more sense as to the 2GB limit I am seeing.

Do I have any control over this? How can I make it bigger? Is this limit coming from OS or board?

Thanks-
Jeff

Right now, our data source has no flow control (hardware handshake), so, yes I am worried about not keeping up and losing data. If the OS has a hard limit of how much can be transferred in one DMA, then, I will have to take it back to the hardware guys and break up the transfer into chunks.

Sorry, I am not quite sure I follow “building multiple partial SGLs out of your giant MDL”. Right now, for my Packet DMA with 58436 byte packets, I am calling GetScatterGatherList once for each packet. When I tried to put PutScatterGatherList/GetScatterGatherList in my DPC, I kept getting BugCheck on GetScatterGatherList. I figured that it was because I was in the wrong context from the buffer alloc context. (buffer virt addresses were wrong)

So now I call GetScatterGatherList on the whole thing up front. And Put… when all done.

Sorry, WdfDmaEnablerGetFragmentLength() gives me back 0x7ffff000 (2GB - PAGE_SIZE) (not 0x7ffff00)…

Jeff,

The OS is giving you a limit when you call WdfDmaEnablerGetFragmentLength – that’s the point of making this call. Even if NTOS *let* you “get away” with a larger transfer, ignoring the limit is a serious architectural error in your driver and driver verifier (you ARE running with Driver Verifier enabled on your test machine, right?? And you’re running it with everything enabled, including DMA verifier, WITH THE EXCEPTION OF Low Resource Simulation???) will hammer you for the violation. You should have both KMDF Verifier and Windows Driver Verifier enabled AT ALL TIME when you’re developing and testing… there’s really no downside to doing this, and LOTS of mistakes will get caught early. OK, I’ll get off my soap-box now. I’m really writing for the archives here, in any case).

In a word, no.

Ah, the question sooo many people ask. The answer is, as far as I know you can’t.

It’s coming from the OS… though, given that you have a 64-bit, device that does scatter/gather, I’m a bit surprised you’re seeing this limit.

Peter
OSR

On 11/04/2010 03:15 AM, xxxxx@osr.com wrote:

In a word, no.

Ah, the question sooo many people ask. The answer is, as far as I know you can’t.

Of course he can:
(1) Buy Microsoft
(2) Order to have this limit changed
(and the change rolled out as a hotfix)

Probably impracticable, but *theoretically* possible. :wink:

Thank you for the help - though I don’t think I will be buying microsoft right yet :wink:

One reason we went to 64 bit platform was to defeat these memory limitations. Win7-64 seems to be giving me the same limitation that XP has…

I am not giving up yet. :wink: There must be something I can look at. Any suggestions???

Peter, your statement

“given that you have a 64-bit, device that does scatter/gather, I’m a bit surprised you’re seeing this limit”

leads me to believe that others have defeated this - now I need to find out what is different b/t theirs and mine…

Is there any clue from the fact that I specify an 8 byte param (size_t) to WDF_DMA_ENABLER_CONFIG_INIT(), but when I ask for that param back via WdfDmaEnablerGetMaximumLength(), it gives me the param back with the top 4 bytes chopped off… I mean, my understanding is that WdfDmaEnablerGetMaximumLength() gives me what I told it, and WdfDmaEnablerGetFragmentLength() gives me the Windows adjusted value.

(I am building in Win7 x64 Checked Build Environment…)

You might have answered this already but is your device on 32 or 64 bit bus. If it is on a 32 bit bus, does it do 64 bit addressing?

–Mark Cariddi
OSR Open Systems Resources, Inc.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@neurologica.com
Sent: Thursday, November 04, 2010 11:42 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] limitations on DMA transfer size?

Thank you for the help - though I don’t think I will be buying microsoft right yet :wink:

One reason we went to 64 bit platform was to defeat these memory limitations. Win7-64 seems to be giving me the same limitation that XP has…

I am not giving up yet. :wink: There must be something I can look at. Any suggestions???

Peter, your statement

“given that you have a 64-bit, device that does scatter/gather, I’m a bit surprised you’re seeing this limit”

leads me to believe that others have defeated this - now I need to find out what is different b/t theirs and mine…

Is there any clue from the fact that I specify an 8 byte param (size_t) to WDF_DMA_ENABLER_CONFIG_INIT(), but when I ask for that param back via WdfDmaEnablerGetMaximumLength(), it gives me the param back with the top 4 bytes chopped off… I mean, my understanding is that WdfDmaEnablerGetMaximumLength() gives me what I told it, and WdfDmaEnablerGetFragmentLength() gives me the Windows adjusted value.

(I am building in Win7 x64 Checked Build Environment…)


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Let’s back up a bit here.

You said:

It’s kinda difficult to believe this is a bug in KMDF… You HAVE the sources to the macros that you reference. Check out the definition of WDF_DMA_ENABLER_CONFIG and what WDF_DMA_ENABLER_CONFIG_INIT does:


typedef struct _WDF_DMA_ENABLER_CONFIG {
//
// Size of this structure in bytes
//
ULONG Size;

//
// One of the above WDF_DMA_PROFILES
//
WDF_DMA_PROFILE Profile;

//
// Maximum DMA Transfer handled in bytes.
//
size_t MaximumLength;

//
// The various DMA PnP/Power event callbacks
//
PFN_WDF_DMA_ENABLER_FILL EvtDmaEnablerFill;
PFN_WDF_DMA_ENABLER_FLUSH EvtDmaEnablerFlush;
PFN_WDF_DMA_ENABLER_DISABLE EvtDmaEnablerDisable;
PFN_WDF_DMA_ENABLER_ENABLE EvtDmaEnablerEnable;
PFN_WDF_DMA_ENABLER_SELFMANAGED_IO_START EvtDmaEnablerSelfManagedIoStart;
PFN_WDF_DMA_ENABLER_SELFMANAGED_IO_STOP EvtDmaEnablerSelfManagedIoStop;

} WDF_DMA_ENABLER_CONFIG, *PWDF_DMA_ENABLER_CONFIG;

VOID
FORCEINLINE
WDF_DMA_ENABLER_CONFIG_INIT(
out PWDF_DMA_ENABLER_CONFIG Config,
in WDF_DMA_PROFILE Profile,
__in size_t MaximumLength
)
{
RtlZeroMemory(Config, sizeof(WDF_DMA_ENABLER_CONFIG));

Config->Size = sizeof(WDF_DMA_ENABLER_CONFIG);
Config->Profile = Profile;
Config->MaximumLength = MaximumLength;
}


Kinda hard for me to believe that there’s a bug there.

And I can confirm the WdfDmaEnablerGetMaximumLength just returns the value from the MaximumLength field of the WDF_DMA_ENABLER_CONFIG object.

Soooo… sounds to me like you’ve got an issue with casting or whatever?

Peter
OSR

Did you using the WDM dma adapter to see of kmdf is giving you problems?

d

dent from a phpne with no keynoard

-----Original Message-----
From: xxxxx@neurologica.com
Sent: November 04, 2010 8:41 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] limitations on DMA transfer size?

Thank you for the help - though I don’t think I will be buying microsoft right yet :wink:

One reason we went to 64 bit platform was to defeat these memory limitations. Win7-64 seems to be giving me the same limitation that XP has…

I am not giving up yet. :wink: There must be something I can look at. Any suggestions???

Peter, your statement

“given that you have a 64-bit, device that does scatter/gather, I’m a bit surprised you’re seeing this limit”

leads me to believe that others have defeated this - now I need to find out what is different b/t theirs and mine…

Is there any clue from the fact that I specify an 8 byte param (size_t) to WDF_DMA_ENABLER_CONFIG_INIT(), but when I ask for that param back via WdfDmaEnablerGetMaximumLength(), it gives me the param back with the top 4 bytes chopped off… I mean, my understanding is that WdfDmaEnablerGetMaximumLength() gives me what I told it, and WdfDmaEnablerGetFragmentLength() gives me the Windows adjusted value.

(I am building in Win7 x64 Checked Build Environment…)


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

The following code:

WDF_DMA_ENABLER_CONFIG_INIT(&dmaConfig,
WdfDmaProfileScatterGather64,
pDmaDevExt->MaximumTransferLength); // set to 0x140000000 or 5GB

status = WdfDmaEnablerCreate(pDevExt->Device,
&dmaConfig,
WDF_NO_OBJECT_ATTRIBUTES,
&pDmaDevExt->DmaEnabler);

if (NT_SUCCESS(status))
{
size_t maxLen = WdfDmaEnablerGetMaximumLength(pDmaDevExt->DmaEnabler);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“dmaConfig.MaximumLength is %lld (0x%x%08x)”, dmaConfig.MaximumLength,
dmaConfig.MaximumLength >> 32,
dmaConfig.MaximumLength);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“WdfDmaEnablerGetMaximumLength returns %lld (0x%x%08x)”,
maxLen, maxLen >> 32, maxLen);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“pDmaDevExt->MaximumTransferLength is %lld (0x%x%08x) (sizeof(size_t) is %d)”,
pDmaDevExt->MaximumTransferLength,
pDmaDevExt->MaximumTransferLength >> 32,
pDmaDevExt->MaximumTransferLength, sizeof(size_t));

… give me the following log entries…

dmaConfig.MaximumLength is 5368709120 (0x140000000)
WdfDmaEnablerGetMaximumLength returns 1073741824 (0x040000000)
pDmaDevExt->MaximumTransferLength is 5368709120 (0x140000000) (sizeof(size_t) is 8)

The only conclusion that I can see from this is that the enabler only took in 4 bytes of the size_t param…

>Did you using the WDM dma adapter to see of kmdf is giving you problems?

I will try this, thanks.

Well, you’ve got ME stumped.

Peter
OSR

xxxxx@neurologica.com wrote:


if (NT_SUCCESS(status))
{
size_t maxLen = WdfDmaEnablerGetMaximumLength(pDmaDevExt->DmaEnabler);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“dmaConfig.MaximumLength is %lld (0x%x%08x)”, dmaConfig.MaximumLength,
dmaConfig.MaximumLength >> 32,
dmaConfig.MaximumLength);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“WdfDmaEnablerGetMaximumLength returns %lld (0x%x%08x)”,
maxLen, maxLen >> 32, maxLen);
TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP,
“pDmaDevExt->MaximumTransferLength is %lld (0x%x%08x) (sizeof(size_t) is %d)”,
pDmaDevExt->MaximumTransferLength,
pDmaDevExt->MaximumTransferLength >> 32,
pDmaDevExt->MaximumTransferLength, sizeof(size_t));

For what it’s worth, you can write 0x%16llx instead of passing the two
halves separately.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

(Those TraceEvents statements are more complex than most drivers I’ve seen :wink:

FWIW, you can write 0x%p and make it even easier.

Peter
OSR

I just tried calling IoGetDmaAdapter() and it is telling me that I have 524288 (0x80000) map registers (corresponding to max transfer length of 2GB). So, this appears to be giving me the same result as KMDF. I gave it a maxLength of 0xfffff000 (4GB - PAGE_SIZE).

>For what it’s worth, you can write 0x%16llx instead of passing the two

halves separately.

(Those TraceEvents statements are more complex than most drivers I’ve seen :wink:

FWIW, you can write 0x%p and make it even easier.

Thanks :stuck_out_tongue:

Your problem is intriguing…

I’d appreciate conformation from some of our friends at Microsoft – like the folks who own the HAL for amd64 these days – but from my brief research it sure looks to ME like the maximum length for a DMA transfer is limited to 2GB by the HAL.

Peter
OSR

“When I tried to put
PutScatterGatherList/GetScatterGatherList in my DPC, I kept getting BugCheck on
GetScatterGatherList. I figured that it was because I was in the wrong context
from the buffer alloc context. (buffer virt addresses were wrong)”

The virtual address in the MDL is used as a reference address if you want a partial SGL. It’s not referenced in any current-process-related way. Because of that, GetSGL should be safe to call in arbitrary DISPATCH_LEVEL context.

The CurrentVa argument of GetSGL is obtained from MmGetMdlVirtualAddress and desired offset. You can call it multiple times for smaller sizes (say, 1GB) and passing varying CurrentVa arguments (say, advancing it by 1GB every time).

Also, keep in mind that GetSGL theoretically (in unusual conditions) may not be synchronous. You should only continue in the callback function.

I wanted to put the question out again, before this thread falls much further into obscurity:

Has anyone done a DMA >2GB on Windows?

Or, more specifically, has anyone been able to specify a MaxTransferSize of >2GB? (to IoGetDmaAdapter() or WdfDmaEnablerCreate())

OK, I guess there’s than one question here… :wink: #3:
Can anyone confirm whether or not there is a 2GB hard limit imposed by OS/HAL??

Thanks
~Jeff