On MDLs

NTDEV
1

Hi, People,

My application requires dma’ing huge 3D voxel buffers. It’s commonplace for
me to have to handle input buffers that consist of 512 slices of 512 x 512
64-bit voxels. That’s 2 raised to 9+9+9+6 = 33, that is, 8 Gigabytes of data
per image buffer! Not exactly your friendly game texture map.

The driver complex provides for dma’ing directly from the user buffer,
which saves a fair amount of performance loss from moving data back and
forth, besides being able to maintain compatibility with Linux and Solaris,
which is a must in my application. I set up an MDL for the user buffer, lock
it, and dma from it. When the interrupt comes, I release the mappings, the
locks and the MDL.

So far so good. The problem is that IoAllocateMdl restricts me to a 64Kb
buffer, and that’s not enough to represent a large and physically fragmented
buffer. When that happens I used to break the buffer into many, but that’s a
royal pain. I recently allowed for the use of the older MDL call, I believe
it’s MmCreateMdl or something like it, but that’s a deprecated call.

Do any of you know of another way to dma from user buffers that doesn’t run
into this problem of overflowing max mdl buffer sizes ? Inquiring minds want
to know. :slight_smile:

Alberto.

2

> The problem is that IoAllocateMdl restricts me to a 64Kb buffer,

What makes you believe so??? Buffer has to be less than PAGE_SIZE * (65535 - sizeof(MDL)) / sizeof(ULONG_PTR), which gives us almost 64MB…

Anton Bassov

3

Hi, Anton,

I actually meant 64Mb, that was a typo - but still, if my memory counts in
gigabytes, every once in a while I get a failure to launch: IoAllocateMdl
fails every once in a while. It’s not that frequent, but it happens. Which
brings the question, what tools do I have to diagnose why exactly the call
to IoAllocateMdl failed ?

I was assuming that it was because the MDL exceeded the max size. But now
you started another train of thought in my mind: maybe what’s happening is
that there’s not enough contiguous space at that point in time to allocate
the MDL. But then, how come things work when I allocate my own buffer ?

Maybe you’re right, what’s happening is something else which I didn’t manage
to put my finger on as yet.

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Saturday, August 25, 2007 2:01 PM
Subject: RE:[ntdev] On MDLs

>> The problem is that IoAllocateMdl restricts me to a 64Kb buffer,
>
> What makes you believe so??? Buffer has to be less than PAGE_SIZE *
> (65535 - sizeof(MDL)) / sizeof(ULONG_PTR), which gives us almost 64MB…
>
> Anton Bassov
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

4

Ahhhh… well, you have WinDbg.

By saying IoAllocateMdl is failing for you, I GUESS you mean it’s returning NULL? So, stop at that point in time and determine:

a) Is there non-paged pool available in the system;
b) Is the buffer you specify larger than 64MB?

The maximum size of the buffer you can describe with an MDL is the maximum size… irrespective of the call you use to build the MDL. Why is it a problem to describe the buffer with multiple MDLs? Seems to me that would be quite straight forward. You MIGHT (hint hint) even be able to chain the MDLs together, if you use the right DMA function calls…

Now that I read your posts again, I guess I’m not really sure what your question is…

Peter
OSR

5

Hi, Peter,

The answers to your question are, (a) yes there must be enough nonpaged pool
available otherwise my direct allocation followed by MmCreateMdl wouldn’t
work, (b) I was blindly assuming that the MDL buffer was larger than 64Mb,
because I was dealing with pretty large data buffers at the time of failure,
and then, (c) I couldn’t make multiple MDLs work, although I confess that I
did this under great pressure and hence I ended up taking the path of least
resistance.

If using multiple MDLs is a feasible path, I’ll try it again! But, mind you,
that’s a royal pain, because while duplicating MDL functionality in Unix is
a piece of cake, chained MDLs begin to add a level of complexity to my Unix
code that I do not want. Because, I forgot to say, this driver is compatible
between Windows, Linux, Solaris and MacOS, and 85% or more of its code is
OS-independent. The code is also totally portable between 32-bit and 64-bit
systems in all of those OS’s.

The question I wanted to ask is this: IoAllocateMdl fails intermittently
with very large user-side data buffers. and as yet I couldn’t figure out
why. When I use MmCreateMdl, the issue goes away and everything works like a
charm!

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Sunday, August 26, 2007 11:04 AM
Subject: RE:[ntdev] On MDLs

>


>
> Ahhhh… well, you have WinDbg.
>
> By saying IoAllocateMdl is failing for you, I GUESS you mean it’s
> returning NULL? So, stop at that point in time and determine:
>
> a) Is there non-paged pool available in the system;
> b) Is the buffer you specify larger than 64MB?
>
> The maximum size of the buffer you can describe with an MDL is the maximum
> size… irrespective of the call you use to build the MDL. Why is it a
> problem to describe the buffer with multiple MDLs? Seems to me that would
> be quite straight forward. You MIGHT (hint hint) even be able to chain
> the MDLs together, if you use the right DMA function calls…
>
> Now that I read your posts again, I guess I’m not really sure what your
> question is…
>
> Peter
> OSR
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

6

Hmmmmm… IoAllocateMdl and MmCreateMdl are indeed separate functions (in other words, this isn’t one of the very common cases where an IoXxxx function merely wraps a function from another subsystem).

IoAllocateMdl will automatically fail requests for MDLs that describe buffers > 2GB… even on x64 systems (which appears to be a mistake to me, but what do I know). This and failure to allocate the MDL from first the lookaside list and then non-paged pool are the only two things that I’m aware of that’ll cause IoAllocateMdl to return NULL.

You can figure out which is causing the failure for you by walking into IoAllocateMdl with the debugger. If it’s the 2GB limit thing, that’ll be obvious pretty quickly.

I dunno… does that help any?

Peter
OSR

7

Yes, Peter, it does help! I will step through it with Windbg and see what I
get. But hey, if 2Gb is the limit of the size of the buffer, that’s probably
the reason, because many of my buffers are larger than that. I was a bit
confused, because if one 4-byte dword can map one 4K page, then I figured
that a 64MB MDL should be able to map a buffer as large (approximately, of
course) as 64Gb. My buffers are big, but not that big, so, here I was,
scratching my head!

I’ll take a look at it tomorrow, if I can make it fail (it’s intermittent,
so, I don’t know a failsafe way of recreating the issue). One of the 3D
graphics guys at the office (by the way, Anton, he’s Russian, his name is
Rostislav, which became “Ross” here in Boston) has an image that causes the
issue pretty frequently, I’ll step through it and I’ll let people know what
I found.

By the way, next time any of you is near Concord, Massachusetts, drop by,
and I can show you some exciting MRI images, such as a revolving brain
aneurysm or a colon flythrough. :slight_smile:

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Sunday, August 26, 2007 12:55 PM
Subject: RE:[ntdev] On MDLs

> Hmmmmm… IoAllocateMdl and MmCreateMdl are indeed separate functions (in
> other words, this isn’t one of the very common cases where an IoXxxx
> function merely wraps a function from another subsystem).
>
> IoAllocateMdl will automatically fail requests for MDLs that describe
> buffers > 2GB… even on x64 systems (which appears to be a mistake to me,
> but what do I know). This and failure to allocate the MDL from first the
> lookaside list and then non-paged pool are the only two things that I’m
> aware of that’ll cause IoAllocateMdl to return NULL.
>
> You can figure out which is causing the failure for you by walking into
> IoAllocateMdl with the debugger. If it’s the 2GB limit thing, that’ll be
> obvious pretty quickly.
>
> I dunno… does that help any?
>
> Peter
> OSR
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

8

Alberto Moreira wrote:

By the way, next time any of you is near Concord, Massachusetts,
drop by, and I can show you some exciting MRI images, such as
a revolving brain aneurysm or a colon flythrough. :slight_smile:

Throw in some Red Sox tickets and you’ve got yourself a deal.

9

Hmm, this could very quickly become a candidate for a new list, “nthumor”…

Somehow the choice of aneurysm and colon flythrough seem tantalizingly
related to kernel development in an odd sort of way. Maybe it’s just me…

-dave

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: Sunday, August 26, 2007 3:13 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] On MDLs

Alberto Moreira wrote:

By the way, next time any of you is near Concord, Massachusetts, drop
by, and I can show you some exciting MRI images, such as a revolving
brain aneurysm or a colon flythrough. :slight_smile:

Throw in some Red Sox tickets and you’ve got yourself a deal.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

10

Hi Alberto,

Is it MR or post-processing station ?
Are you sure you really need DMA all the slices at once ?

Regards,
Igor

11

Hi, Igor,

This is a medical imaging system, it’s called “Aquarius Net”. A typical
application might have a server and several remote clients; the server has
the boards and it has services that handle the multiple aspects of
rendering, up to submitting a candidate image to the chip. The client allows
remote stations, for example, at a Doctor’s office, to store and retrieve
medical images at the server. The images come from such equipment as MRI
scanners.

The services send the raw unprocessed images to the chip. Together with the
image, the services also send multiple rendering tables and specialized
command streams to the driver, so as to do the kind of rendering they need.
I get those images, tables and commands through dma into on-board memory,
from which the chips perform the actual rendering. The chips are also able
to pull data directly from host memory.

I have no control over how those slices look like or how they’re organized;
the application handles all the multiprocessing control and segmentation,
and typically it passes me those images, or parts thereof. They have several
volume rendering algorithms that split images this way and that way, and a
lot of thought is given to keep all those processors busy and operating in
parallel, so as to increase the frame rate! Moreover, the company supplies
the volume rendering functionality as a dll within a volume rendering SDK,
so that customers can add their own visualization applications; the driver
supports that by maintaining a pretty high level interface to the outside
world.

So, I have to transact whatever the API gives me!

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Monday, August 27, 2007 12:37 PM
Subject: RE:[ntdev] On MDLs

> Hi Alberto,
>
> Is it MR or post-processing station ?
> Are you sure you really need DMA all the slices at once ?
>
> Regards,
> Igor
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

12

Hi Alberto,

Thanks for the explanations.
A few thoughts. If all the rendering made by the special board - may you query not for the full buffer but only the 2D image for display ? In case if you need the whole 3D buffer is not easier transfer it by the pieces ? In this case you can cancel the non-completed transfer by the user request.

Igor

p.s.By the way - I have some cancer XRays and CTs for the exchange :wink:

13

Hi, Igor,

The raw image must get to the board first. The chip applies a number of
volume rendering and signal processing algorithms to convert the raw data
into an image. It’s those input images that are huge. Once we render, the
output image is a normal 1280x1024 or 1600x1200 planar image, which is much
easier to manipulate! But to get the input data into the chip, that’s what
takes a lot of bandwidth.

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Tuesday, August 28, 2007 11:17 AM
Subject: RE:[ntdev] On MDLs

> Hi Alberto,
>
> Thanks for the explanations.
> A few thoughts. If all the rendering made by the special board - may you
> query not for the full buffer but only the 2D image for display ? In case
> if you need the whole 3D buffer is not easier transfer it by the pieces ?
> In this case you can cancel the non-completed transfer by the user
> request.
>
> Igor
>
> p.s.By the way - I have some cancer XRays and CTs for the exchange :wink:
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

14

Oh gee … I’ve had my clolon flown through a time or two … and I can’t
say I’m really interested in seeing it in living 3D. :slight_smile:

–
The personal opinion of
Gary G. Little

“Alberto Moreira” wrote in message news:xxxxx@ntdev…
> Yes, Peter, it does help! I will step through it with Windbg and see what
> I get. But hey, if 2Gb is the limit of the size of the buffer, that’s
> probably the reason, because many of my buffers are larger than that. I
> was a bit confused, because if one 4-byte dword can map one 4K page, then
> I figured that a 64MB MDL should be able to map a buffer as large
> (approximately, of course) as 64Gb. My buffers are big, but not that big,
> so, here I was, scratching my head!
>
> I’ll take a look at it tomorrow, if I can make it fail (it’s intermittent,
> so, I don’t know a failsafe way of recreating the issue). One of the 3D
> graphics guys at the office (by the way, Anton, he’s Russian, his name is
> Rostislav, which became “Ross” here in Boston) has an image that causes
> the issue pretty frequently, I’ll step through it and I’ll let people know
> what I found.
>
> By the way, next time any of you is near Concord, Massachusetts, drop by,
> and I can show you some exciting MRI images, such as a revolving brain
> aneurysm or a colon flythrough. :slight_smile:
>
>
> Alberto.
>
>
> ----- Original Message -----
> From:
> To: “Windows System Software Devs Interest List”
> Sent: Sunday, August 26, 2007 12:55 PM
> Subject: RE:[ntdev] On MDLs
>
>
>> Hmmmmm… IoAllocateMdl and MmCreateMdl are indeed separate functions (in
>> other words, this isn’t one of the very common cases where an IoXxxx
>> function merely wraps a function from another subsystem).
>>
>> IoAllocateMdl will automatically fail requests for MDLs that describe
>> buffers > 2GB… even on x64 systems (which appears to be a mistake to
>> me, but what do I know). This and failure to allocate the MDL from first
>> the lookaside list and then non-paged pool are the only two things that
>> I’m aware of that’ll cause IoAllocateMdl to return NULL.
>>
>> You can figure out which is causing the failure for you by walking into
>> IoAllocateMdl with the debugger. If it’s the 2GB limit thing, that’ll be
>> obvious pretty quickly.
>>
>> I dunno… does that help any?
>>
>> Peter
>> OSR
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
>