About MmAllocatePagesForMdl for allocating over then 4 GB

NTDEV
1

Hello,
?
I am developing a RAMDISK driver for x64 bit OS. I have?12 GB RAM installed on PC and OS (Vista x64) can detect this amount of RAM.
I am using MmAllocatePagesForMdl or MmAllocatePagesForMdlEx function for allocating the memory. My driver is build as an x64.

I have another application? which is x32 bit compatible. My application is passing the user data to the driver. Also I have testing using only the driver without the application.

Now my problem is: MmAllocatePagesForMdl or MmAllocatePagesForMdlEx
function always giving 3.95 GB even I have?12 GB ram installed. According to my
long search in internet, I didn’t find any clue. This function should work as my understanding.

I am stack here, If some one has the similar problem that already solved, or have the knowledge please help me.

Here is the main section of code for allocating the memory.

#define GB_SIZE 5
#define MB_SIZE GB_SIZE*1024
#define RAM_SIZE_BYTE MB_SIZE*1024*1024

//Other codes

PHYSICAL_ADDRESS lowAddress;
PHYSICAL_ADDRESS highAddress;
SIZE_T totalBytes;

lowAddress.QuadPart = 0;
highAddress.QuadPart = 0x200000000; // 8 GB
totalBytes = RAM_SIZE_BYTE;
?
highAddress.QuadPart = ( highAddress.QuadPart / PAGE_SIZE ) * PAGE_SIZE;

?
lpDevExt->pMdl = MmAllocatePagesForMdlEx( lowAddress, highAddress, lowAddress, totalBytes,MmNonCached,0);
?
if ( lpDevExt->pMdl == NULL )
{
??? ramDiskCleanUp( lpFunctionDeviceObject );
??? return STATUS_INSUFFICIENT_RESOURCES;
}

2

> #define GB_SIZE 5

#define MB_SIZE GB_SIZE*1024
#define RAM_SIZE_BYTE MB_SIZE*1024*1024

I believe this results in a computing constant overflow. Change GB_SIZE to 5ULL

highAddress.QuadPart = 0x200000000; // 8 GB

Why the 8GB cap, doesn’t that greatly constrict the size? Here again I think it is good practice to use ULL anywhere constants are larger than 4 bytes.

3

Dear Sven,

Thank you for your quick reply.

I have took a look according to your suggestions.

Basically I update the code like below, for sure result.

I believe this results in a computing constant overflow. Change GB_SIZE to 5ULL

> I rewrite it as
unsigned __int64 lRamDiskSize = 0x140000000ULL;

Why the 8GB cap, doesn’t that greatly constrict the size?

> Yes, I keep the size constrict only for testing purpose. I just want to
allocate 5GB out of 8GB ram. I will change it later ofcourse.

But still I could not get the expected result. In Debug View the lRamdiskSize value shows as 5368709120 which equals to 0x140000000ULL;

Also my highAddress value is shown as 8589934592 = 0x200000000

So I think, now the value is OK after change according to your suggestion.
But the RAMDISK created with always 3.96 GB now.
BTW I formatted the Ramdisk in FAT32 file system.

Is there any other reason for not able to allocated 5 GB RAM here, for example ?

Looking forward to your answers.

Best Regards
-Mostafa

— On Thu, 5/28/09, xxxxx@gmail.com wrote:

From: xxxxx@gmail.com
Subject: RE:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB
To: “Windows System Software Devs Interest List”
Date: Thursday, May 28, 2009, 7:10 PM

> #define GB_SIZE 5
> #define MB_SIZE GB_SIZE1024
> #define RAM_SIZE_BYTE MB_SIZE1024*1024

I believe this results in a computing constant overflow. Change GB_SIZE to 5ULL

> highAddress.QuadPart = 0x200000000; // 8 GB

Why the 8GB cap, doesn’t that greatly constrict the size? Here again I think it is good practice to use ULL anywhere constants are larger than 4 bytes.

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

4

Another Information is,

My MmGetMdlByteCount(…) function return the value 4294963200 = 0xFFFFF000
which is almost 3.99 GB.

— On Fri, 5/29/09, James Rassel wrote:

From: James Rassel
Subject: RE:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB
To: “Windows System Software Devs Interest List”
Date: Friday, May 29, 2009, 7:54 AM

Dear Sven,

Thank you for your quick reply.

I have took a look according to your suggestions.

Basically I update the code like below, for sure result.

I believe this results in a computing constant overflow. Change GB_SIZE to 5ULL
>> I rewrite it as
unsigned __int64 lRamDiskSize = 0x140000000ULL;

Why the 8GB cap, doesn’t that greatly constrict the size?
>> Yes, I keep the size constrict only for testing purpose. I just want to
allocate 5GB out of 8GB ram. I will change it later ofcourse.

But still I could not get the expected result. In Debug View the lRamdiskSize value shows as 5368709120 which equals to 0x140000000ULL;

Also my highAddress value is shown as 8589934592 = 0x200000000

So I think, now the value is OK after change according to your suggestion.
But the RAMDISK created
with always 3.96 GB now.
BTW I formatted the Ramdisk in FAT32 file system.

Is there any other reason for not able to allocated 5 GB RAM here, for example ?

Looking forward to your answers.

Best Regards
-Mostafa

— On Thu, 5/28/09, xxxxx@gmail.com wrote:

From: xxxxx@gmail.com
Subject: RE:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB
To: “Windows System Software Devs Interest List”
Date: Thursday, May 28, 2009, 7:10 PM

> #define GB_SIZE 5
> #define MB_SIZE GB_SIZE1024
> #define RAM_SIZE_BYTE MB_SIZE1024*1024

I believe this results in a computing constant overflow. Change GB_SIZE to 5ULL

>
highAddress.QuadPart = 0x200000000; // 8 GB

Why the 8GB cap, doesn’t that greatly constrict the size? Here again I think it is good practice to use ULL anywhere constants are larger than 4 bytes.

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

5

> Now my problem is: MmAllocatePagesForMdl or MmAllocatePagesForMdlEx

function always giving 3.95 GB even I have 12 GB ram installed.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE).

Note that even much smaller allocations can fail in practice, even if the
total
amount of RAM is greater than 4 GB. Typically, MmAllocatePagesForMdlEx
can only allocate as much as the amount of available pages in the system
(Memory\Available Bytes counter in perfmon or “Available” memory
in task manager).

–
Pavel Lebedinsky/Windows Kernel Test
This posting is provided “AS IS” with no warranties, and confers no rights.

6

Dear Mr. Pavel,

Thank you for your reply.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE)

> So I am confirm that, it is not possible to create over then 4 GB for a single call
using these functions. What about the Multiple call and uses the allocated memory
as a single RAM drive ? Is there any possibilities ? Like using some Array of MDL.

e.g, PMDL pMdl[10];

Sorry for asking some child-like questions, I don’t have much experience on device driver,
but I really need to create over 4 GB RAM (e.x 10 GB, 20 GB etc) for a single RAM disk drive. If someone can give me some clue, I can study forward.
But still I can find any clue to solve my problem.

Anyway thanks again and looking forward for your advice.

BR
-Mostafa

— On Fri, 5/29/09, Pavel Lebedinsky wrote:

From: Pavel Lebedinsky
Subject: Re:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB
To: “Windows System Software Devs Interest List”
Date: Friday, May 29, 2009, 1:00 PM

> Now my problem is: MmAllocatePagesForMdl or MmAllocatePagesForMdlEx
> function always giving 3.95 GB even I have 12 GB ram installed.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE).

Note that even much smaller allocations can fail in practice, even if the
total
amount of RAM is greater than 4 GB. Typically, MmAllocatePagesForMdlEx
can only allocate as much as the amount of available pages in the system
(Memory\Available Bytes counter in perfmon or “Available” memory
in task manager).

–
Pavel Lebedinsky/Windows Kernel Test
This posting is provided “AS IS” with no warranties, and confers no rights.

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

7

Use multiple allocations?

  • S

From: James Rassel
Sent: Friday, May 29, 2009 06:21
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB

Dear Mr. Pavel,

Thank you for your reply.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE)

>> So I am confirm that, it is not possible to create over then 4 GB for a single call
using these functions. What about the Multiple call and uses the allocated memory
as a single RAM drive ? Is there any possibilities ? Like using some Array of MDL.

e.g, PMDL pMdl[10];

Sorry for asking some child-like questions, I don’t have much experience on device driver,
but I really need to create over 4 GB RAM (e.x 10 GB, 20 GB etc) for a single RAM disk drive. If someone can give me some clue, I can study forward.
But still I can find any clue to solve my problem.

Anyway thanks again and looking forward for your advice.

BR
-Mostafa

— On Fri, 5/29/09, Pavel Lebedinsky wrote:

From: Pavel Lebedinsky
Subject: Re:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB
To: “Windows System Software Devs Interest List”
Date: Friday, May 29, 2009, 1:00 PM

> Now my problem is: MmAllocatePagesForMdl or MmAllocatePagesForMdlEx
> function always giving 3.95 GB even I have 12 GB ram installed.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE).

Note that even much smaller allocations can fail in practice, even if the
total
amount of RAM is greater than 4 GB. Typically, MmAllocatePagesForMdlEx
can only allocate as much as the amount of available pages in the system
(Memory\Available Bytes counter in perfmon or “Available” memory
in task manager).

–
Pavel Lebedinsky/Windows Kernel Test
This posting is provided “AS IS” with no warranties, and confers no rights.

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

8

James Rassel wrote:

Sorry for asking some child-like questions, I don’t have much
experience on device driver,
but I really need to create over 4 GB RAM (e.x 10 GB, 20 GB etc) for a
single RAM disk drive. If someone can give me some clue, I can study
forward.
But still I can find any clue to solve my problem.

Why do you want to do this?

RAM drives are a fun toy, and they were useful in the “bad old days”
when we were working with floppy drives that worked at the speed of
snail, but they have very little practical use today. Windows disk
caching produces a more useful speed boost, and doesn’t require you to
think about a separate drive letter.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

9

Why do you need the memory to be virtually contiguous? You could allocate in much smaller slabs and use a pretty simple tree data structure to find your way through the various memory pages. The larger the slab you try to allocate, the more fragile your ramdisk driver will become.

-p

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of James Rassel
Sent: Friday, May 29, 2009 3:26 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB

Dear Mr. Pavel,

Thank you for your reply.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE)

> So I am confirm that, it is not possible to create over then 4 GB for a single call
using these functions. What about the Multiple call and uses the allocated memory
as a single RAM drive ? Is there any possibilities ? Like using some Array of MDL.

e.g, PMDL pMdl[10];

Sorry for asking some child-like questions, I don’t have much experience on device driver,
but I really need to create over 4 GB RAM (e.x 10 GB, 20 GB etc) for a single RAM disk drive. If someone can give me some clue, I can study forward.
But still I can find any clue to solve my problem.

Anyway thanks again and looking forward for your advice.

BR
-Mostafa

— On Fri, 5/29/09, Pavel Lebedinsky > wrote:

From: Pavel Lebedinsky >
Subject: Re:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB
To: “Windows System Software Devs Interest List” >
Date: Friday, May 29, 2009, 1:00 PM
> Now my problem is: MmAllocatePagesForMdl or MmAllocatePagesForMdlEx
> function always giving 3.95 GB even I have 12 GB ram installed.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE).

Note that even much smaller allocations can fail in practice, even if the
total
amount of RAM is greater than 4 GB. Typically, MmAllocatePagesForMdlEx
can only allocate as much as the amount of available pages in the system
(Memory\Available Bytes counter in perfmon or “Available” memory
in task manager).

–
Pavel Lebedinsky/Windows Kernel Test
This posting is provided “AS IS” with no warranties, and confers no rights.

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

10

Thank you all for the comments and ideas.
Here I comment back.

**********************
**** Mr. peterwie ****
**********************

Why do you need the memory to be virtually contiguous??
You could allocate in much smaller slabs and use a pretty
simple tree data structure to find your way through the various memory pages.?
The larger the slab you try to allocate, the more fragile your ramdisk driver will become.

MY COMMENT:

According to the documentation, MmAllocatePagesForMdl or MmAllocatePagesForMdlEx doesn't
allocate memory in contigous form, so my idea was to allocate the LARGE memory.
It is written as

"The physical memory pages returned by MmAllocatePagesForMdl
are typically not contiguous pages"

in Microsoft Learn: Build skills that open doors in your career

Are you meaning different thing by "virtually contiguous" ?

I couldn't get why "fragile". Could you please explain more about it ?

*************************
**** Mr. Tim Roberts ****
*************************

Why do you want to do this?

RAM drives are a fun toy, and they were useful in the "bad old days"
when we were working with floppy drives that worked at the speed of
snail, but they have very little practical use today.? Windows disk
caching produces a more useful speed boost, and doesn't require you to
think about a separate drive letter.

MY COMMENT:

Thanks for your comments...

"bad old days", nice phrase,
BUT...I think some popular commercial Ramdisk software are still living
in "bad old days", OR , may be the users like bad old days.

There are several reasons for us to make such drive.
A seperate drive is required in our SPEC, so we couldn't
change it.

********************
**** Mr Skywing ****
********************

Use multiple allocations?

MY COMMENTS

Sorry I couldn't get your point.
Basically I can create , for example 8 GB in Ram, with 4 GB each.
So even I use multiple application, I need to make 2 drives for 4GB each.
But I need only one drive with 8GB.

Do you have other thinking...?

Best Regards
-M

--- On Fri, 5/29/09, Peter Wieland wrote:

From: Peter Wieland
Subject: RE: Re:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB
To: "Windows System Software Devs Interest List"
Date: Friday, May 29, 2009, 11:35 PM

Why do you need the memory to be virtually contiguous?? You could allocate in much smaller slabs and use a pretty simple tree data structure to find your way through the various memory pages.? The larger the slab you try to allocate, the more fragile your ramdisk driver will become. ?-p ?From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of James Rassel
Sent: Friday, May 29, 2009 3:26 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB ?Dear Mr. Pavel,

Thank you for your reply.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE)

>> So I am confirm that, it is not possible to create over then 4 GB for a single call
using these functions. What about the Multiple call and uses the allocated memory
as a single RAM drive ? Is there any possibilities ? Like using some Array of MDL.

e.g, PMDL pMdl[10];

Sorry for asking some child-like questions, I don't have much experience on device driver,
but I really need to create over 4 GB RAM (e.x 10 GB, 20 GB etc) for a single RAM disk drive. If someone can give me some clue, I can study forward.
But still I can find any clue to solve my problem.

Anyway thanks again and looking forward for your advice.

BR
-Mostafa

--- On Fri, 5/29/09, Pavel Lebedinsky wrote:
From: Pavel Lebedinsky
Subject: Re:[ntdev] About MmAllocatePagesForMdl for allocating over then 4 GB
To: "Windows System Software Devs Interest List"
Date: Friday, May 29, 2009, 1:00 PM> Now my problem is: MmAllocatePagesForMdl or MmAllocatePagesForMdlEx
> function always giving 3.95 GB even I have 12 GB ram installed.

Maximum MDL size is determined by the ByteCount field which is a ULONG.
So the largest size that can be allocated in a single call to
MmAllocatePagesForMdl
is (4 GB - PAGE_SIZE).

Note that even much smaller allocations can fail in practice, even if the
total
amount of RAM is greater than 4 GB. Typically, MmAllocatePagesForMdlEx
can only allocate as much as the amount of available pages in the system
(Memory\Available Bytes counter in perfmon or "Available" memory
in task manager).

--
Pavel Lebedinsky/Windows Kernel Test
This posting is provided "AS IS" with no warranties, and confers no rights.

---
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
OSR Seminars – OSR

To unsubscribe, visit the List Server section of OSR Online at ListServer/Forum
--- NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: OSR Seminars – OSR To unsubscribe, visit the List Server section of OSR Online at ListServer/Forum

---

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:

OSR Seminars – OSR

To unsubscribe, visit the List Server section of OSR Online at ListServer/Forum

11

>So even I use multiple application, I need to make 2 drives for 4GB each.

But I need only one drive with 8GB.

Implement a single drive which will be based on several underlying MDLs, as Peter said.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

12

Actually my point is here, can not I use multiple MDL using a single application?

Sent from my iPhone

On May 31, 2009, at 12:03 PM, “Maxim S. Shatskih” wrote:

So even I use multiple application, I need to make 2 drives for 4GB each.
But I need only one drive with 8GB.

Implement a single drive which will be based on several underlying MDLs, as Peter said.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

13

Dude, maybe you should go back to the store where you bought your iPhone and buy a shrink wrapped RAMDrive for like $30.

Good luck,

mm

14

Yahoo wrote:

Actually my point is here, can not I use multiple MDL using a single application?

I don’t know what you mean by “application” here. Several people have
repeatedly advised you to allocate several blocks of memory. Make a
number of smaller allocations until you have the amount you need, then
store your data in these different areas of memory. If for example you
allocate 8 pieces of memory of 1 GB each to implement an 8 GB virtual
disk, then you just need to implement some trivial algorithms to convert
a sector number in the virtual disk into an offset in a particular
block of memory.

15

James Rassel wrote:

According to the documentation, MmAllocatePagesForMdl or
MmAllocatePagesForMdlEx doesn’t
allocate memory in contigous form, so my idea was to allocate the
LARGE memory.
It is written as

“The physical memory pages returned by MmAllocatePagesForMdl
are typically not contiguous pages”

in http://msdn.microsoft.com/en-us/library/ms802812.aspx

Are you meaning different thing by “virtually contiguous” ?

Absolutely. You know the difference between virtual addresses and
physical addresses, right? A buffer allocated by any of the generic
memory allocation routines will always be contiguous in virtual memory,
but most of them will not be contiguous in physical memory.

The occasions for which you need physically contiguous memory are rare
and rather specialized, almost always involving DMA. If you are not
using DMA, then you don’t care about physical addresses at all.

If you had thought about it for a moment, you’d realize that it would be
silly for a memory allocator to return a buffer that was not virtually
contiguous.

I couldn’t get why “fragile”. Could you please explain more about it ?

Because after the operating system has run for a little while, it
becomes impossible to allocate large chunks of physically contiguous
memory. Physical address space quickly gets fragmented, and there’s no
good way to compact it.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.