Allocating memory in kernel mode

I want my driver to work with a big amount of memory, which is very fast (not paged out). What is the best way to allocate this memory? It can be 500 Mb for Windows XP x32. Using of paged or nonpaged pool under x32-versions is not good, because both are limited to ~256 MB and other drivers need them too. Ok, under x64 I can just use ExAllocatePoolWithTag. I read another thread and wrote this test function:

bool AllocateAndLock(char*& Allocated, SIZE_T MEM_SIZE)
{
PMDL mdl;
Allocated = (char*)MmAllocateNonCachedMemory(MEM_SIZE);
//Allocated = (char*)ExAllocatePoolWithTag(PagedPool, MEM_SIZE, 0x44564D20);
if (Allocated == NULL)
{
DBG_TL(“MmAllocateNonCachedMemory failed \n”);
return false;
}

mdl = IoAllocateMdl(Allocated, (ULONG)MEM_SIZE, FALSE, FALSE, 0);
if (mdl == 0)
{
DBG_TL(“IoAllocateMdl failed \n”);
MmFreeNonCachedMemory(Allocated, MEM_SIZE);
//ExFreePoolWithTag(Allocated, 0x44564D20);
Allocated = NULL;
return false;
}
//ULONG ull = MmGetMdlByteCount(mdl);
//DBG_TVL("MEM_SIZE == ", (u64)MEM_SIZE);
//DBG_TVL("ull == ", (u64)ull);
//MmBuildMdlForNonPagedPool(mdl);

MmProbeAndLockPages(mdl, KernelMode, IoWriteAccess);

return true;
}

I hoped, MmProbeAndLockPages() would make the allocated buffer unoupageable and fast, but it is actually slower as a buffer in paged pool allocated with ExAllocatePoolWithTag even without MmProbeAndLockPages(). How can I reserve a big amount of memory not from paged or nonpaged pool in kernel mode and make it unoutpageable?

Or is it easier to allocate it in user mode and give an adress to the driver for locking?

Non-cached memory means the CPU never uses the onboard cache, and always goes out on the main memory bus. You do want to allocate memory and probe and lock it, but not non-cached memory. If you really need to allocate massive amounts of memory in the driver, and make it non-paged, you have almost got it right, you just need a different allocation routine.

I would ask you to take a step back and question the need for allocating that much memory in the driver. If you were to tell us the problem you are trying to solve, we might suggest a better way than this.

Phil

Philip D Barila

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmx.de
Sent: Thursday, February 09, 2012 8:07 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Allocating memory in kernel mode

I want my driver to work with a big amount of memory, which is very fast (not paged out). What is the best way to allocate this memory? It can be 500 Mb for Windows XP x32. Using of paged or nonpaged pool under x32-versions is not good, because both are limited to ~256 MB and other drivers need them too. Ok, under x64 I can just use ExAllocatePoolWithTag. I read another thread and wrote this test function:

bool AllocateAndLock(char*& Allocated, SIZE_T MEM_SIZE)
{
PMDL mdl;
Allocated = (char*)MmAllocateNonCachedMemory(MEM_SIZE);
//Allocated = (char*)ExAllocatePoolWithTag(PagedPool, MEM_SIZE, 0x44564D20);
if (Allocated == NULL)
{
DBG_TL(“MmAllocateNonCachedMemory failed \n”);
return false;
}

mdl = IoAllocateMdl(Allocated, (ULONG)MEM_SIZE, FALSE, FALSE, 0);
if (mdl == 0)
{
DBG_TL(“IoAllocateMdl failed \n”);
MmFreeNonCachedMemory(Allocated, MEM_SIZE);
//ExFreePoolWithTag(Allocated, 0x44564D20);
Allocated = NULL;
return false;
}
//ULONG ull = MmGetMdlByteCount(mdl);
//DBG_TVL("MEM_SIZE == ", (u64)MEM_SIZE);
//DBG_TVL("ull == ", (u64)ull);
//MmBuildMdlForNonPagedPool(mdl);

MmProbeAndLockPages(mdl, KernelMode, IoWriteAccess);

return true;
}

I hoped, MmProbeAndLockPages() would make the allocated buffer unoupageable and fast, but it is actually slower as a buffer in paged pool allocated with ExAllocatePoolWithTag even without MmProbeAndLockPages(). How can I reserve a big amount of memory not from paged or nonpaged pool in kernel mode and make it unoutpageable?

Or is it easier to allocate it in user mode and give an adress to the driver for locking?


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

You do realize you are taking roughly one quarter to one half of the
address space in the kernel in a 32-bit system for your effort. If this
is a dedicated system that can potentially be ok, but otherwise this is
way out of line. The reason your performance stinks is you specified
non-cached memory.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@gmx.de” wrote in message
news:xxxxx@ntdev:

> I want my driver to work with a big amount of memory, which is very fast (not paged out). What is the best way to allocate this memory? It can be 500 Mb for Windows XP x32. Using of paged or nonpaged pool under x32-versions is not good, because both are limited to ~256 MB and other drivers need them too. Ok, under x64 I can just use ExAllocatePoolWithTag. I read another thread and wrote this test function:
>
> bool AllocateAndLock(char*& Allocated, SIZE_T MEM_SIZE)
> {
> PMDL mdl;
> Allocated = (char*)MmAllocateNonCachedMemory(MEM_SIZE);
> //Allocated = (char*)ExAllocatePoolWithTag(PagedPool, MEM_SIZE, 0x44564D20);
> if (Allocated == NULL)
> {
> DBG_TL(“MmAllocateNonCachedMemory failed \n”);
> return false;
> }
>
> mdl = IoAllocateMdl(Allocated, (ULONG)MEM_SIZE, FALSE, FALSE, 0);
> if (mdl == 0)
> {
> DBG_TL(“IoAllocateMdl failed \n”);
> MmFreeNonCachedMemory(Allocated, MEM_SIZE);
> //ExFreePoolWithTag(Allocated, 0x44564D20);
> Allocated = NULL;
> return false;
> }
> //ULONG ull = MmGetMdlByteCount(mdl);
> //DBG_TVL("MEM_SIZE == ", (u64)MEM_SIZE);
> //DBG_TVL("ull == ", (u64)ull);
> //MmBuildMdlForNonPagedPool(mdl);
>
> MmProbeAndLockPages(mdl, KernelMode, IoWriteAccess);
>
> return true;
> }
>
> I hoped, MmProbeAndLockPages() would make the allocated buffer unoupageable and fast, but it is actually slower as a buffer in paged pool allocated with ExAllocatePoolWithTag even without MmProbeAndLockPages(). How can I reserve a big amount of memory not from paged or nonpaged pool in kernel mode and make it unoutpageable?
>
> Or is it easier to allocate it in user mode and give an adress to the driver for locking?

It’s a kind of RAM-disk functionality I try to test, so it must be big enough to keep user’s data. The driver should apply the SCSI-commands on memory instead of hard disk. But if a RAM-disk will be paged out to the hard disk by system, it makes no sense…

That is really a bad idea. There has been a ton of ramdisk efforts and
none of them produce performance to justify them versus just using the
windows file system caching. In recent system your ramdisk is taking
memory away from the cache, and on 32-bit system you are restricting
memory enough that you are causing more overhead from paging, in both
cases it is likely you have degraded system performance.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@gmx.de” wrote in message
news:xxxxx@ntdev:

> It’s a kind of RAM-disk functionality I try to test, so it must be big enough to keep user’s data. The driver should apply the SCSI-commands on memory instead of hard disk. But if a RAM-disk will be paged out to the hard disk by system, it makes no sense…

xxxxx@gmx.de wrote:

It’s a kind of RAM-disk functionality I try to test, so it must be big enough to keep user’s data. The driver should apply the SCSI-commands on memory instead of hard disk. But if a RAM-disk will be paged out to the hard disk by system, it makes no sense…

Oh, of course it does. This is only for testing, right? RAM disks in
production situations are a bad idea.

The memory will only be paged out if the pages are not being
referenced. In that case, it is a complete waste of resources to tie up
valuable RAM with unused pages. Just let the page manager do its
thing. That assumes, of course, that your driver will always be running
at a passive IRQL.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

MmAllocatePagesForMdl


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
>I want my driver to work with a big amount of memory, which is very fast (not paged out). What is the best way to allocate this memory? It can be 500 Mb for Windows XP x32. Using of paged or nonpaged pool under x32-versions is not good, because both are limited to ~256 MB and other drivers need them too. Ok, under x64 I can just use ExAllocatePoolWithTag. I read another thread and wrote this test function:
>
> bool AllocateAndLock(char*& Allocated, SIZE_T MEM_SIZE)
> {
> PMDL mdl;
> Allocated = (char*)MmAllocateNonCachedMemory(MEM_SIZE);
> //Allocated = (char*)ExAllocatePoolWithTag(PagedPool, MEM_SIZE, 0x44564D20);
> if (Allocated == NULL)
> {
> DBG_TL(“MmAllocateNonCachedMemory failed \n”);
> return false;
> }
>
> mdl = IoAllocateMdl(Allocated, (ULONG)MEM_SIZE, FALSE, FALSE, 0);
> if (mdl == 0)
> {
> DBG_TL(“IoAllocateMdl failed \n”);
> MmFreeNonCachedMemory(Allocated, MEM_SIZE);
> //ExFreePoolWithTag(Allocated, 0x44564D20);
> Allocated = NULL;
> return false;
> }
> //ULONG ull = MmGetMdlByteCount(mdl);
> //DBG_TVL("MEM_SIZE == ", (u64)MEM_SIZE);
> //DBG_TVL("ull == ", (u64)ull);
> //MmBuildMdlForNonPagedPool(mdl);
>
> MmProbeAndLockPages(mdl, KernelMode, IoWriteAccess);
>
> return true;
> }
>
> I hoped, MmProbeAndLockPages() would make the allocated buffer unoupageable and fast, but it is actually slower as a buffer in paged pool allocated with ExAllocatePoolWithTag even without MmProbeAndLockPages(). How can I reserve a big amount of memory not from paged or nonpaged pool in kernel mode and make it unoutpageable?
>
> Or is it easier to allocate it in user mode and give an adress to the driver for locking?
>
>

RAMdisks are popular because under MS-DOS they improved performance.
Windows is not MS-DOS, and why do you think a RAMdisk will provide any
performance improvement? You probably have zero data on this, and as Don
points out, if you can get the space at all, you will profoundly impact
overall system performance. And why in the world would you think that you
have to acquire it in a single allocation? That’s just begging for
failure! You would be best served by dropping the idea completely.

Back on MS-DOS, I had a database program that ran for five hours. By
copying the files to an MS-DOS RAMdisk, it ran in 40 minutes. On my
Windows machine, without RAMdisk, it runs in under five minutes. (You see,
I DO have data).
joe

That is really a bad idea. There has been a ton of ramdisk efforts and
none of them produce performance to justify them versus just using the
windows file system caching. In recent system your ramdisk is taking
memory away from the cache, and on 32-bit system you are restricting
memory enough that you are causing more overhead from paging, in both
cases it is likely you have degraded system performance.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@gmx.de” wrote in message
> news:xxxxx@ntdev:
>
>> It’s a kind of RAM-disk functionality I try to test, so it must be big
>> enough to keep user’s data. The driver should apply the SCSI-commands on
>> memory instead of hard disk. But if a RAM-disk will be paged out to the
>> hard disk by system, it makes no sense…
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

RAM-disk for windows is a nice toy. Users love it, because they want to control everything. RAM-disks are good for test of special software and file systems.
Well, from technical point of view, windows page manager is probably the best ram-disk. Generally. But it doesn’t know, which files the user intends to access soon. If there are no use cases under windows at all, in which extern RAM-disks bring better performance - I don’t know, but I wouldn’t stick my neck out for that.

The only use cases I have seen are for special configurations i.e. not a
general purpose computer. And even then that was back in the Windows
2000 days and the cache has been improved significantly since then.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@gmx.de” wrote in message
news:xxxxx@ntdev:

> RAM-disk for windows is a nice toy. Users love it, because they want to control everything. RAM-disks are good for test of special software and file systems.
> Well, from technical point of view, windows page manager is probably the best ram-disk. Generally. But it doesn’t know, which files the user intends to access soon. If there are no use cases under windows at all, in which extern RAM-disks bring better performance - I don’t know, but I wouldn’t stick my neck out for that.

> RAM-disk for windows is a nice toy. Users love it, because they want to control everything. RAM-

disks are good for test of special software and file systems.

VHDMP embedded to Windows 7+ is even a better thing for such.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

Hi,

I’ve almost got it. The one problem I have - I can’t use a PMDL returned by MmAllocatePagesForMdl() for a call of MmProbeAndLockPages(). An exception with code C0000005h (access violation) is raised.
But I can allocate another PMDL with IoAllocateMdl and lock pages with this another PMDL.
The problem is - IoAllocateMdl() fails to allocate mdl if the size reserved by MmAllocatePagesForMdl() is 100 Mb (maybe 64 Mb is th limit here). It’s not understandable - some functions ending with Mdl work actually with chains of MDL, other work with singular MDL. Is there an analog of IoAllocateMdl() which would return a chain of MDLs, which can be used for MmProbeAndLockPages()?

No there is no MDL chain support for MmProbeAndLockPages and MDL’s do
have physical limits of a little less than 64MB in XP. You have to do
this with multiple MDL’s and using the address parameter to
MmProbeAndLockPages.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@gmx.de” wrote in message
news:xxxxx@ntdev:

> Hi,
>
> I’ve almost got it. The one problem I have - I can’t use a PMDL returned by MmAllocatePagesForMdl() for a call of MmProbeAndLockPages(). An exception with code C0000005h (access violation) is raised.
> But I can allocate another PMDL with IoAllocateMdl and lock pages with this another PMDL.
> The problem is - IoAllocateMdl() fails to allocate mdl if the size reserved by MmAllocatePagesForMdl() is 100 Mb (maybe 64 Mb is th limit here). It’s not understandable - some functions ending with Mdl work actually with chains of MDL, other work with singular MDL. Is there an analog of IoAllocateMdl() which would return a chain of MDLs, which can be used for MmProbeAndLockPages()?

MmAllocatePagesForMdl returns an MDL that’s already locked so you don’t need to probe-and-lock it (which, as you discovered, wouldn’t work anyway, because MmProbeAndLockPages expects an MDL with a valid virtual address, and MmAllocatePagesForMdl sets Mdl->StartVa to NULL).

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmx.de
Sent: Wednesday, February 15, 2012 3:56 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Allocating memory in kernel mode

Hi,

I’ve almost got it. The one problem I have - I can’t use a PMDL returned by MmAllocatePagesForMdl() for a call of MmProbeAndLockPages(). An exception with code C0000005h (access violation) is raised.
But I can allocate another PMDL with IoAllocateMdl and lock pages with this another PMDL.
The problem is - IoAllocateMdl() fails to allocate mdl if the size reserved by MmAllocatePagesForMdl() is 100 Mb (maybe 64 Mb is th limit here). It’s not understandable - some functions ending with Mdl work actually with chains of MDL, other work with singular MDL. Is there an analog of IoAllocateMdl() which would return a chain of MDLs, which can be used for MmProbeAndLockPages()?


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Thank you, Pavel, it’s an important point. :slight_smile:

So, it works now, but allocated memory seems not to be released correctly. I call

MmFreePagesFromMdl(pmdl);
ExFreePool(pmdl);

for pmdl created by MmAllocatePagesForMdl().

The memory seems to be released in the task manager - the same amount of physical memory is available as before allocating. But if I try to call MmAllocatePagesForMdl() again, the pmdl is created, but MmGetSystemAddressForMdlSafe(pmdl, NormalPagePriority) returns NULL. MmGetMdlVirtualAddress(pmdl) returns also NULL. MmGetMdlByteCount(pmdl) returns the correct size in bytes requested by allocation. I must restart the system to allocate memory once more.
Or if I allocate a smaller amount, N allocations can be done, N = (free memory)/(allocated amount), before this behaviour occurs. What is wrong with my releasing calls or the way I get the virtual address of the buffer?


MmAllocatePagesForMdl returns an MDL that’s already locked so you don’t need to probe-and-lock it (which, as you discovered, wouldn’t work anyway, because MmProbeAndLockPages expects an MDL with a valid virtual address, and MmAllocatePagesForMdl sets Mdl->StartVa to NULL).

I found out that MmUnmapLockedPages() has to be called before releasing. More allocations can be done now, but the problem still exists. Maybe it’s another error.

> I’ve almost got it. The one problem I have - I can’t use a PMDL returned by MmAllocatePagesForMdl

for a call of MmProbeAndLockPages()

This is normal. Pages from MmAllocatePagesForMdl are already memory-resident.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

> ExFreePool(pmdl);

Do IoFreeMdl instead, it will do the proper unmap and thus release the system PTEs which you’re leaking.

Or - the same - do MmPrepareMdlForReuse+ExFreePool.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com