MmAllocateMappingAddress() and MmProbeAndLockPages()

[Bill_Alexander]

I’m trying to manage my own virtual address space from my driver, but I am having trouble when try to allocate, map and lock a subset of the VA space. Here is what I am doing:

// Allocate our virtual address space
// Start at 1 TB and work our way down
va_space_size = MAX_VA_SPACE_SIZE;
do
{
va_space = MmAllocateMappingAddress (
va_space_size,
PGALLCTR_TAG);
if ( !va_space )
{
// Negoiate to smaller size
va_space_size >>= 1;
if ( va_space_size < MIN_VA_SPACE_SIZE )
{
error (“Cannot allocate VA space!\n”);
goto Exit;
}
}
} while (!va_space);

> I actually get 64 GB of VA space allocated – which is plenty for my
> purposes

> I tried two different ways to allocate an MLD

// Allocate an MDL for the slot address
alloc_size = 128 * 1024 * 1024;
//mdl = IoAllocateMdl (ptr, alloc_size, FALSE, FALSE, NULL);
//if ( !mdl )
//{
// error (“IoAllcateMdl failed\n”);
// goto Exit;
//}
low.QuadPart = 0;
high.QuadPart = 0xFFFFFFFFFFFFFFFF;
skip.QuadPart = PAGE_SIZE;
mdl = MmAllocatePagesForMdlEx (low, high, skip, alloc_size,
MmNonCached, 0);
if ( !mdl )
{
error (“MmAllocatePagesForMdlEx failed\n”);
goto Exit;
}

> I don’t know if I need this following line

MmBuildMdlForNonPagedPool (mdl);

> When I try to probe and lock, MmProbeAndLockPages() throws an
> EXCEPTION_EXECUTION_HANDLER exception with status 0xC0000005,
> which is an access violation.

__try
{
// Probe and lock pages
MmProbeAndLockPages (mdl, KernelMode, IoWriteAccess);
locked = TRUE;
}
__except(EXCEPTION_EXECUTE_HANDLER)
{
error (“MmProbeAndLockPages failed, status = 0x%x\n”,
GetExceptionCode());
goto Exit;
}

I would appreciate any help or reference.

thx,

((&->

[Mark_Roddy]

MmMapLockedPagesWithReservedMapping - the docs claim this is the api to use
after using MmAllocateMappingAddress.

Mark Roddy

On Tue, Sep 14, 2010 at 10:25 AM, wrote:

> I’m trying to manage my own virtual address space from my driver, but I am
> having trouble when try to allocate, map and lock a subset of the VA space.
> Here is what I am doing:
>
> // Allocate our virtual address space
> // Start at 1 TB and work our way down
> va_space_size = MAX_VA_SPACE_SIZE;
> do
> {
> va_space = MmAllocateMappingAddress (
> va_space_size,
> PGALLCTR_TAG);
> if ( !va_space )
> {
> // Negoiate to smaller size
> va_space_size >>= 1;
> if ( va_space_size < MIN_VA_SPACE_SIZE )
> {
> error (“Cannot allocate VA space!\n”);
> goto Exit;
> }
> }
> } while (!va_space);
>
> >> I actually get 64 GB of VA space allocated – which is plenty for my
> >> purposes
>
>
> >> I tried two different ways to allocate an MLD
>
> // Allocate an MDL for the slot address
> alloc_size = 128 * 1024 * 1024;
> //mdl = IoAllocateMdl (ptr, alloc_size, FALSE, FALSE, NULL);
> //if ( !mdl )
> //{
> // error (“IoAllcateMdl failed\n”);
> // goto Exit;
> //}
> low.QuadPart = 0;
> high.QuadPart = 0xFFFFFFFFFFFFFFFF;
> skip.QuadPart = PAGE_SIZE;
> mdl = MmAllocatePagesForMdlEx (low, high, skip, alloc_size,
> MmNonCached, 0);
> if ( !mdl )
> {
> error (“MmAllocatePagesForMdlEx failed\n”);
> goto Exit;
> }
>
> >> I don’t know if I need this following line
>
> MmBuildMdlForNonPagedPool (mdl);
>
> >> When I try to probe and lock, MmProbeAndLockPages() throws an
> >> EXCEPTION_EXECUTION_HANDLER exception with status 0xC0000005,
> >> which is an access violation.
>
> __try
> {
> // Probe and lock pages
> MmProbeAndLockPages (mdl, KernelMode, IoWriteAccess);
> locked = TRUE;
> }
>__except(EXCEPTION_EXECUTE_HANDLER)
> {
> error (“MmProbeAndLockPages failed, status = 0x%x\n”,
> GetExceptionCode());
> goto Exit;
> }
>
>
> I would appreciate any help or reference.
>
> thx,
>
> ((&->
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[Bill_Alexander]

True, but docs also say that you have to call IoAllocateMdl() and MmProbeAndLockPages() before calling MmMapLockedPagesWithReservedMapping(). I quote:

*********************************************************************

The caller can use MmMapLockedPagesWithReservedMapping to map a subrange of the virtual memory range reserved by MmAllocateMappingAddress as follows:

1. Use IoAllocateMdl to allocate an MDL. The returned MDL is built using the specified starting address and size of the subrange of the virtual memory range to map.

2. Use MmProbeAndLockPages to lock down the physical pages described by the MDL obtained in step 1.

3. Use MmMapLockedPagesWithReservedMapping to actually map the virtual memory to the physical memory that was locked down in step 2. Note that the virtual address returned by this function does include the byte offset that the MDL specifies. However, the MappedSystemVa field of the MDL that is set by this function does not include the byte offset.

*********************************************************************

So what am I missing? Perhaps some parameter to IoAllocateMdl()? An example would be very helpful.

((&->

[Mark_Roddy]

IoAllocateMdl is not going to allocate an MDL for a size greater than a
little bit less than 64mb. MDLs are more or less limited to describing
locked pages that are in this range.

Mark Roddy

On Tue, Sep 14, 2010 at 11:47 AM, wrote:

> True, but docs also say that you have to call IoAllocateMdl() and
> MmProbeAndLockPages() before calling MmMapLockedPagesWithReservedMapping().
> I quote:
>
>
>
>
> The caller can use MmMapLockedPagesWithReservedMapping to map a subrange of
> the virtual memory range reserved by MmAllocateMappingAddress as follows:
>
> 1. Use IoAllocateMdl to allocate an MDL. The returned MDL is built
> using the specified starting address and size of the subrange of the virtual
> memory range to map.
>
> 2. Use MmProbeAndLockPages to lock down the physical pages described
> by the MDL obtained in step 1.
>
> 3. Use MmMapLockedPagesWithReservedMapping to actually map the
> virtual memory to the physical memory that was locked down in step 2. Note
> that the virtual address returned by this function does include the byte
> offset that the MDL specifies. However, the MappedSystemVa field of the MDL
> that is set by this function does not include the byte offset.
>
>
>
> So what am I missing? Perhaps some parameter to IoAllocateMdl()? An
> example would be very helpful.
>
> ((&->
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[Peter_Viscarola_OSR]

IoAllocateMdl is not going to allocate an MDL for a size greater than a
little bit less than 64mb

Your comment that the max size of an MDL was 64Mb surprised me… so I spent time this morning looking at the code for IoAllocateMdl and MmMapLockedPagesWithReservedMapping to try to determine if this limit still exists.

I couldn’t see the limit you mentioned anywhere. Then Mark Cariddi pointed out the latest doc page on IoAllocateMdl:

http://msdn.microsoft.com/en-us/library/ff548263(VS.85).aspx

The Length parameter specifies the size of the buffer that is to be described by the MDL. In Windows Server 2003, Windows XP, and Windows 2000, the maximum buffer size, in bytes, that this routine can allocate is PAGE_SIZE * (65535 - sizeof(MDL)) / sizeof(ULONG_PTR). In Windows Vista and Windows Server 2008, the maximum buffer size is (2 gigabytes - PAGE_SIZE). In Windows 7 and Windows Server 2008 R2, the maximum buffer size is (4 gigabytes - PAGE_SIZE).

Actually, it looks to ME like the actual maximum size allowed depends on the function used to create the MDL, but in no case do I see the maximum being less than 2GB (on Vista and later).

Peter
OSR

[Mark_Roddy]

Right. Being a dinosaur, I tend to view things from XP’s operating mode. So
for 86% or so of the installed base…

Mark Roddy

On Tue, Sep 14, 2010 at 12:53 PM, wrote:

>

> IoAllocateMdl is not going to allocate an MDL for a size greater than a
> little bit less than 64mb
>

>
> Your comment that the max size of an MDL was 64Mb surprised me… so I
> spent time this morning looking at the code for IoAllocateMdl and
> MmMapLockedPagesWithReservedMapping to try to determine if this limit still
> exists.
>
> I couldn’t see the limit you mentioned anywhere. Then Mark Cariddi pointed
> out the latest doc page on IoAllocateMdl:
>
> http://msdn.microsoft.com/en-us/library/ff548263(VS.85).aspx
>
>

> The Length parameter specifies the size of the buffer that is to be
> described by the MDL. In Windows Server 2003, Windows XP, and Windows 2000,
> the maximum buffer size, in bytes, that this routine can allocate is
> PAGE_SIZE * (65535 - sizeof(MDL)) / sizeof(ULONG_PTR). In Windows Vista and
> Windows Server 2008, the maximum buffer size is (2 gigabytes - PAGE_SIZE).
> In Windows 7 and Windows Server 2008 R2, the maximum buffer size is (4
> gigabytes - PAGE_SIZE).
>

>
> Actually, it looks to ME like the actual maximum size allowed depends on
> the function used to create the MDL, but in no case do I see the maximum
> being less than 2GB (on Vista and later).
>
> Peter
> OSR
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[Peter_Viscarola_OSR]

(Well, it’s nice that they finally documented this limitation, so we can point people to something in the docs.

Peter
OSR)

[Bill_Alexander]

Thanks for info, guys.

I think I know what the problem is.

I need to call ExAllocatePoolWithTag() to allocate the actual physical memory. This was the intent of MmAllocatePagesForMdlEx(), but I think that is not working the way I think it is supposed to work.

So the new plan is to call ExAllocatePoolWithTag() and then call IoAllocateMdl() to get an MDL for the mapping. I probably don’t need to call MmProbeAndLockPages(), since ExAllocatePoolWithTag() has already locked the pages in non-paged memory.

I’ll let you know what happens.

((&->

[Peter_Viscarola_OSR]

I need to call ExAllocatePoolWithTag() to allocate the actual physical memory.

No… you really do NOT want to do that.

Non-paged MEMORY in Windows IS NOT the same as non-paged pool (this is a common mistake).

So… MmAllocatePagesForMdlEx was what you want, if what you want is to allocate non-paged pages of memory. Those pages are already probed and locked. So, your next step should be to call MmMapLockedPagesWithReservedMapping.

You see, it’s actually EASIER than you think it is…

Peter
OSR

[Bill_Alexander]

Yep - you were correct. It worked.

So to recap for the audience:

1. Call MmAllocateMappingAddress() to reserve the VA space.
2. Call MmAllocatePagesForMdlEx() to allocate non-paged physical pages and lock them.
3. Call MmMapLockedPagesWithReservedMapping() to map the physical pages into the VA space.

Now I need to map multiple chunks into the VA space returned in step 1 above. It is a little confusing, since the doc for MmMapLockedPagesWithReservedMapping() says you have to pass the VA start address which was obtained from MmAllocateMappingAddress():

******************************************************************
MappingAddress [in]

Pointer to the beginning of the reserved virtual memory range. This must be an address previously returned by MmAllocateMappingAddress.

******************************************************************

There appears to be no way to specify an offset from the start of the VA space, since MmAllocatePagesForMdlEx() does not let you specify an offset.

Perhaps there is a function to set the offset in the MDL after calling MmAllocatePagesForMdlEx() – you can tell I’m not an MDL expert

((&->

[Bill_Alexander]

Looks like MmAdvanceMdl() is what I need.

[Peter_Wieland]

From talking to the owner of MM in the past my understanding is that it’s very rare for MmAdvanceMdl to be what you want. Among other things it modifies the original MDL, unlocks and releases pages, etc… It’s for pretty specialized uses.

The purpose of a reserve address is to map ONE MDL, not to map a whole bunch of different pages.

So why are you trying to build your own memory manager inside the existing one?

-p

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@Yahoo.com
Sent: Tuesday, September 14, 2010 12:29 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MmAllocateMappingAddress() and MmProbeAndLockPages()

Looks like MmAdvanceMdl() is what I need.

---

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[Pavel_Lebedinsky]

Right, there is no way to map multiple MDLs (or chunks of the same MDL) into a single reserved mapping region. MmMapLockedPagesWithReservedMapping should bugcheck if you try this.

Thanks,
Pavel

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Peter Wieland
Sent: Tuesday, September 14, 2010 2:19 PM
To: Windows System Software Devs Interest List
Subject: RE: RE:[ntdev] MmAllocateMappingAddress() and MmProbeAndLockPages()

From talking to the owner of MM in the past my understanding is that it’s very rare for MmAdvanceMdl to be what you want. Among other things it modifies the original MDL, unlocks and releases pages, etc… It’s for pretty specialized uses.

The purpose of a reserve address is to map ONE MDL, not to map a whole bunch of different pages.

So why are you trying to build your own memory manager inside the existing one?

[Alex_Grig]

Yes, what’s the ultimate purpose of the whole exercise? What problem you’re trying to solve?

[Bill_Alexander]

The problem is that WS03 np memory deallocation is slow.

We have a data structure that grows one page at a time over time to about 10 GB. That is 2.6 million pages. When we shutdown, it takes over 7 hours to free all of the pages - that is 10 ms per ExFreePoolWithTag() call.

So I was researching the possibility of doing slab management and suballocating pages from there. The idea was to reserve a large VA space using AllocMappingAddr() – say 64 GB. And then allocating 128 MB slabs from the VA space using MapLockedPages(). When freeing a page, we could easily find the slab using pointer arithmetic from the beginning of the VA space, since each slot starts on a 128 MB boundary.

Sounds good in theory, but it appears you can only do one Map from the VA at a time. Dang! Another one bites the dust.

Our original idea was to alloc 128 MB slabs using ExAllocationPoolWithTag() and then linking the slabs in a linked list. But when you go to free a page, you have to traverse the linked list looking the the slab that contains the page. That’s a pain. (Anybody have a better idea?)

BTW, Pavel was right. MmMapLockedPagesWithReservedMapping() does bugcheck when it is called again with the same VA space.

((&->

[Alex_Grig]

Just do ExAllocatePool with larger pieces (like 1MB or 4MB). I see you don’t need it physically contiguous or anything. Then do your suballoc from those pieces.

[Bill_Alexander]

Yeah, that was the original idea. But freeing a page within a slab is kinda slow because you have to traverse through a linked list of slabs. That’s were the VA space idea came in – to make freeing a page very fast.

[Peter_Viscarola_OSR]

The problem is that WS03 np memory deallocation is slow.

We have a data structure that grows one page at a time over time to about 10 GB.
That is 2.6 million pages. When we shutdown, it takes over 7 hours to free all
of the pages - that is 10 ms per ExFreePoolWithTag() call.

Well… hmmm… the question is WHY it’s slow. Off the top of my head, I can’t think of anything that would intrinsically CAUSE this to be slow… ESPECIALLY if you’re using Non-paged pool… unless you’re allocating really small chunks.

Did you do any profiling to see WHERE the time is being spent? It always seems to me that it makes sense to analyze first, implement second, you know?

You’ll need to be careful about allocating the memory yourself, mapping it, and then unmapping it. I can think of SEVERAL reasons why THIS might be slow… for example, when you change a virtual to physical mapping there’ll need to be a TLB flush on all processors in the system. MM is pretty cool about trying to batch up these invalidates, but still… it incurs a significant system-wide perf penalty.

Peter
OSR

[Alex_Grig]

>But freeing a page within a slab is kinda
slow because you have to traverse through a linked list of slabs.

Do you ever need to shrink your structure back or it only grows?
If you don’t need to shrink it, why bother with freeing single pages?

In any case, finding a slab by an address of included page is O(log2(n)) complexity, where n is number of slabs.

[Bill_Alexander]

The structure does shrink and performance with ExAllocatePoolWithTag()/ExFreePoolWithTag() is very good. However, in the extreme case that I am trying to address, we wanted to get really good performance.

A simple linked list will probably be good enough. With 10 GB, I calculate about 80 slabs (at 128 MB each) will be needed. A simple binary tree might just do the trick to speed it up over the a linear linked list. However, with only 80 nodes, a simple array of pointers may be even faster to look up. The thing is, I need to make the slab size configurable. If the slab size is 4 MB, that gives me 2,560 slabs to store 10 GB. Both a linked list look up and a linear array scan would be horrid.

In the case of 2,560 slabs, a tree (or binary search on an array) would offer much better performance with log2(2560) (11.3) maximum search. I don’t remember the formula for ave search, but assuming the ave is half of the max, then that is 5.15, which is pretty good.