Mapping arbitrary PFN's in userspace

NTFSD
1

Hello All,

I have a slightly strange requirement. I am in a hypervisor environment
where I receive a bunch of contiguous guest PFN’s. These translate to
arbitrary PFN’s on the host (Windows 7). I need a way to map these
arbitrary PFN’s into user space of a process in the context of a
DeviceIoControl.

So far, I have been unable to find a documented way to construct a Mdl by
specifying arbitrary PFN’s. If it was possible, I would have been able to
map it via MmMapLockedPagesSpecifyCache.

Just now, I am mapping each of these PFN’s one by one using MmMapIoSpace,
constructing an Mdl for that page and then mapping it into user space by
reserving some prior user space range and unmapping it page by page. This
approach is just for testing purposes, but it has some serious problems.

Is there a way to do it cleanly i.e. is there a simple way to construct an
Mdl by specifying a bunch of PFN’s (which are not contiguous?). Are there
any examples of people doing something like this by hand?

Thanks
screwgauge

2

There is no clean way of doing this. If you look at the routine
MmAllocatePagesForMdlEx for example, it shows you the necessary steps to
build up an MDL with a range of PFNs.

I would not recommend this approach for any release code.

Pete

On 8/8/2013 7:33 PM, Suman Saraf wrote:

Hello All,

I have a slightly strange requirement. I am in a hypervisor environment
where I receive a bunch of contiguous guest PFN’s. These translate to
arbitrary PFN’s on the host (Windows 7). I need a way to map these
arbitrary PFN’s into user space of a process in the context of a
DeviceIoControl.

So far, I have been unable to find a documented way to construct a Mdl
by specifying arbitrary PFN’s. If it was possible, I would have been
able to map it via MmMapLockedPagesSpecifyCache.

Just now, I am mapping each of these PFN’s one by one using
MmMapIoSpace, constructing an Mdl for that page and then mapping it into
user space by reserving some prior user space range and unmapping it
page by page. This approach is just for testing purposes, but it has
some serious problems.

Is there a way to do it cleanly i.e. is there a simple way to construct
an Mdl by specifying a bunch of PFN’s (which are not contiguous?). Are
there any examples of people doing something like this by hand?

Thanks
screwgauge

— NTFSD is sponsored by OSR OSR is hiring!! Info at
http://www.osr.com/careers For our schedule of debugging and file system
seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295

3

Why do you need to map any kernel pages into user space? This is
generally a design error that arises because someone wants to make Windows
work like some embedded RTOS once seen by the questioner.

Here’s a very simple rule: if your design requires mapping kernel pages
into user space, reconsider the design.

Do NOT be driven from the bottom up. This way lies frustration,
heartbreak, and disaster. Instead, pay more attention to how you can map
user pages into the kernel. You will find yourself with a much easier
problem.

And before you run away, saying, “I could do that, but it would require an
inefficient copy operation”, please pay attention to the fact that we are
no longer running on PDP-11s or 8088s. We are running with gigahertz
clocks on machines with pipelining of both instructions and data, an
asynchronous opportunistic execution engine, and multilevel caches.
Unless you have actual performance numbers on the cost of a copy, you have
no rationale for declaring it “inefficient”.

When you ask questions like this, you must not omit critical data that
makes a sensible answer possible. How many bytes do you need to copy?
How many times per second do you need to do this? A “gut feeling” about
performance is not data. Measured performance data is data. Quantities
are crucial. You asked an undefined question involving a problematic
solution that has several catastrophic failure modes if not done
PERFECTLY. As an engineer, you need to assess the Return On Investment of
a fragile, complex, and quite possibly unsupportable solution that may
take weeks or months to get “right enough” to ship (at which point,
unfortunately, you begin to discover the not-inconsiderable gulf between
“right enough” and “actually right”, which tends to annoy your customers),
or a simple, straightforward, supported, and robust interface that
violates your “gut feeling” about performance in the absence of any
supporting data.

It may be that for the number of terabytes per second you have to transfer
that copying IS too slow. But all we see here is a request to explain te
best way to pluck feathers so you can glue them to your pig, without
explaining the nature of the bird, the size of the pig, or why you think a
flying pig actually solves your problem.

There is no clean way of doing this. If you look at the routine
MmAllocatePagesForMdlEx for example, it shows you the necessary steps to
build up an MDL with a range of PFNs.

I would not recommend this approach for any release code.

Pete

On 8/8/2013 7:33 PM, Suman Saraf wrote:
> Hello All,
>
> I have a slightly strange requirement. I am in a hypervisor environment
> where I receive a bunch of contiguous guest PFN’s. These translate to
> arbitrary PFN’s on the host (Windows 7). I need a way to map these
> arbitrary PFN’s into user space of a process in the context of a
> DeviceIoControl.
>
> So far, I have been unable to find a documented way to construct a Mdl
> by specifying arbitrary PFN’s. If it was possible, I would have been
> able to map it via MmMapLockedPagesSpecifyCache.
>
> Just now, I am mapping each of these PFN’s one by one using
> MmMapIoSpace, constructing an Mdl for that page and then mapping it into
> user space by reserving some prior user space range and unmapping it
> page by page. This approach is just for testing purposes, but it has
> some serious problems.
>
> Is there a way to do it cleanly i.e. is there a simple way to construct
> an Mdl by specifying a bunch of PFN’s (which are not contiguous?). Are
> there any examples of people doing something like this by hand?
>
> Thanks
> screwgauge
>
> — NTFSD is sponsored by OSR OSR is hiring!! Info at
> http://www.osr.com/careers For our schedule of debugging and file system
> seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
> List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

–
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295

NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

4

I need to set up a mapping between a user process in host and a user
process in guest in my own proprietary hypervisor environment. One of the
ways to do it is with hypervisor assistance and that is exactly what I am
doing. These pages are not Windows kernel pages. And yes, I am doing this
to implement zero-copy. And I require a throughput of roughly 1Gbps on
large copies of graphics textures.

Needless to say, this is not the first stab at this problem. Sufficient
benchmarking has established memcpy’s as the bottleneck in our current
solution.

Thanks for your philosophical answer and design inputs without pointing to
any plausible answers.

Thank you.

screwgauge

On Fri, Aug 9, 2013 at 1:52 PM, wrote:

> Why do you need to map any kernel pages into user space? This is
> generally a design error that arises because someone wants to make Windows
> work like some embedded RTOS once seen by the questioner.
>
> Here’s a very simple rule: if your design requires mapping kernel pages
> into user space, reconsider the design.
>
> Do NOT be driven from the bottom up. This way lies frustration,
> heartbreak, and disaster. Instead, pay more attention to how you can map
> user pages into the kernel. You will find yourself with a much easier
> problem.
>
> And before you run away, saying, “I could do that, but it would require an
> inefficient copy operation”, please pay attention to the fact that we are
> no longer running on PDP-11s or 8088s. We are running with gigahertz
> clocks on machines with pipelining of both instructions and data, an
> asynchronous opportunistic execution engine, and multilevel caches.
> Unless you have actual performance numbers on the cost of a copy, you have
> no rationale for declaring it “inefficient”.
>
> When you ask questions like this, you must not omit critical data that
> makes a sensible answer possible. How many bytes do you need to copy?
> How many times per second do you need to do this? A “gut feeling” about
> performance is not data. Measured performance data is data. Quantities
> are crucial. You asked an undefined question involving a problematic
> solution that has several catastrophic failure modes if not done
> PERFECTLY. As an engineer, you need to assess the Return On Investment of
> a fragile, complex, and quite possibly unsupportable solution that may
> take weeks or months to get “right enough” to ship (at which point,
> unfortunately, you begin to discover the not-inconsiderable gulf between
> “right enough” and “actually right”, which tends to annoy your customers),
> or a simple, straightforward, supported, and robust interface that
> violates your “gut feeling” about performance in the absence of any
> supporting data.
>
> It may be that for the number of terabytes per second you have to transfer
> that copying IS too slow. But all we see here is a request to explain te
> best way to pluck feathers so you can glue them to your pig, without
> explaining the nature of the bird, the size of the pig, or why you think a
> flying pig actually solves your problem.
>
> >
> > There is no clean way of doing this. If you look at the routine
> > MmAllocatePagesForMdlEx for example, it shows you the necessary steps to
> > build up an MDL with a range of PFNs.
> >
> > I would not recommend this approach for any release code.
> >
> > Pete
> >
> > On 8/8/2013 7:33 PM, Suman Saraf wrote:
> >> Hello All,
> >>
> >> I have a slightly strange requirement. I am in a hypervisor environment
> >> where I receive a bunch of contiguous guest PFN’s. These translate to
> >> arbitrary PFN’s on the host (Windows 7). I need a way to map these
> >> arbitrary PFN’s into user space of a process in the context of a
> >> DeviceIoControl.
> >>
> >> So far, I have been unable to find a documented way to construct a Mdl
> >> by specifying arbitrary PFN’s. If it was possible, I would have been
> >> able to map it via MmMapLockedPagesSpecifyCache.
> >>
> >> Just now, I am mapping each of these PFN’s one by one using
> >> MmMapIoSpace, constructing an Mdl for that page and then mapping it into
> >> user space by reserving some prior user space range and unmapping it
> >> page by page. This approach is just for testing purposes, but it has
> >> some serious problems.
> >>
> >> Is there a way to do it cleanly i.e. is there a simple way to construct
> >> an Mdl by specifying a bunch of PFN’s (which are not contiguous?). Are
> >> there any examples of people doing something like this by hand?
> >>
> >> Thanks
> >> screwgauge
> >>
> >> — NTFSD is sponsored by OSR OSR is hiring!! Info at
> >> http://www.osr.com/careers For our schedule of debugging and file
> system
> >> seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
> >> List Server section of OSR Online at
> >> http://www.osronline.com/page.cfm?name=ListServer
> >
> > –
> > Kernel Drivers
> > Windows File System and Device Driver Consulting
> > www.KernelDrivers.com
> > 866.263.9295
> >
> > —
> > NTFSD is sponsored by OSR
> >
> > OSR is hiring!! Info at http://www.osr.com/careers
> >
> > For our schedule of debugging and file system seminars visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
> >
>
>
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

5

> I need to set up a mapping between a user process in host and a user

process in guest in my own proprietary hypervisor environment. One of the
ways to do it is with hypervisor assistance and that is exactly what I am
doing. These pages are not Windows kernel pages. And yes, I am doing this
to implement zero-copy. And I require a throughput of roughly 1Gbps on
large copies of graphics textures.

Yes, I agree, given the data rate you need, the best solution to share
pages. There are ways of accomplishing this, but I’m no longer sure of
all the magic required. But now that you’ve given the parameters of the
problem, the chances of getting a useful answer have increased. There are
some ugly details, such as what happens if the host process or guest
process terminate, or guest machine terminates, etc., but I’d suggest
looking into the idea of memory-mapped files. If you have a file that can
be shared between the host file system and the guest file system, try
having the two processes involved open the file with shared read/write
access, then use the file mapping API. It may be that this will work (I’m
not precisely sure if the kernel can handle this across host/guest, but
the experts can weigh in on this). Perhaps you only need to tweak some
aspect of this kind of sharing might be something that might be simpler
than a complex mapping, but I’m speculating well outside my comfort zone.
But instead of using PFNs, which are undocumented, and change based on a
ton of parameters (does the CPU support more that nGB of RAM; does it have
PAE or not; how much physical memory is installed; what was the phase of
the moon when the fab masks were created for the CPU; etc.) so they would
be very risky, subject to arbitrary changes. Wht you probably want to do
is take the physical address (a 64-bit number even on 32-bit processors,
to allow for the maximum 64GB of RAM they can support), and map it to a
virtual address in the guest context. The PFNs you create might work on
your machine, but not on mine. Or even on yours, after the next set of
hotfixes come down.
joe

Needless to say, this is not the first stab at this problem. Sufficient
benchmarking has established memcpy’s as the bottleneck in our current
solution.

Thanks for your philosophical answer and design inputs without pointing to
any plausible answers.

Thank you.

screwgauge

On Fri, Aug 9, 2013 at 1:52 PM, wrote:
>
>> Why do you need to map any kernel pages into user space? This is
>> generally a design error that arises because someone wants to make
>> Windows
>> work like some embedded RTOS once seen by the questioner.
>>
>> Here’s a very simple rule: if your design requires mapping kernel pages
>> into user space, reconsider the design.
>>
>> Do NOT be driven from the bottom up. This way lies frustration,
>> heartbreak, and disaster. Instead, pay more attention to how you can
>> map
>> user pages into the kernel. You will find yourself with a much easier
>> problem.
>>
>> And before you run away, saying, “I could do that, but it would require
>> an
>> inefficient copy operation”, please pay attention to the fact that we
>> are
>> no longer running on PDP-11s or 8088s. We are running with gigahertz
>> clocks on machines with pipelining of both instructions and data, an
>> asynchronous opportunistic execution engine, and multilevel caches.
>> Unless you have actual performance numbers on the cost of a copy, you
>> have
>> no rationale for declaring it “inefficient”.
>>
>> When you ask questions like this, you must not omit critical data that
>> makes a sensible answer possible. How many bytes do you need to copy?
>> How many times per second do you need to do this? A “gut feeling” about
>> performance is not data. Measured performance data is data. Quantities
>> are crucial. You asked an undefined question involving a problematic
>> solution that has several catastrophic failure modes if not done
>> PERFECTLY. As an engineer, you need to assess the Return On Investment
>> of
>> a fragile, complex, and quite possibly unsupportable solution that may
>> take weeks or months to get “right enough” to ship (at which point,
>> unfortunately, you begin to discover the not-inconsiderable gulf between
>> “right enough” and “actually right”, which tends to annoy your
>> customers),
>> or a simple, straightforward, supported, and robust interface that
>> violates your “gut feeling” about performance in the absence of any
>> supporting data.
>>
>> It may be that for the number of terabytes per second you have to
>> transfer
>> that copying IS too slow. But all we see here is a request to explain
>> te
>> best way to pluck feathers so you can glue them to your pig, without
>> explaining the nature of the bird, the size of the pig, or why you think
>> a
>> flying pig actually solves your problem.
>>
>> >
>> > There is no clean way of doing this. If you look at the routine
>> > MmAllocatePagesForMdlEx for example, it shows you the necessary steps
>> to
>> > build up an MDL with a range of PFNs.
>> >
>> > I would not recommend this approach for any release code.
>> >
>> > Pete
>> >
>> > On 8/8/2013 7:33 PM, Suman Saraf wrote:
>> >> Hello All,
>> >>
>> >> I have a slightly strange requirement. I am in a hypervisor
>> environment
>> >> where I receive a bunch of contiguous guest PFN’s. These translate to
>> >> arbitrary PFN’s on the host (Windows 7). I need a way to map these
>> >> arbitrary PFN’s into user space of a process in the context of a
>> >> DeviceIoControl.
>> >>
>> >> So far, I have been unable to find a documented way to construct a
>> Mdl
>> >> by specifying arbitrary PFN’s. If it was possible, I would have been
>> >> able to map it via MmMapLockedPagesSpecifyCache.
>> >>
>> >> Just now, I am mapping each of these PFN’s one by one using
>> >> MmMapIoSpace, constructing an Mdl for that page and then mapping it
>> into
>> >> user space by reserving some prior user space range and unmapping it
>> >> page by page. This approach is just for testing purposes, but it has
>> >> some serious problems.
>> >>
>> >> Is there a way to do it cleanly i.e. is there a simple way to
>> construct
>> >> an Mdl by specifying a bunch of PFN’s (which are not contiguous?).
>> Are
>> >> there any examples of people doing something like this by hand?
>> >>
>> >> Thanks
>> >> screwgauge
>> >>
>> >> — NTFSD is sponsored by OSR OSR is hiring!! Info at
>> >> http://www.osr.com/careers For our schedule of debugging and file
>> system
>> >> seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
>> >> List Server section of OSR Online at
>> >> http://www.osronline.com/page.cfm?name=ListServer
>> >
>> > –
>> > Kernel Drivers
>> > Windows File System and Device Driver Consulting
>> > www.KernelDrivers.com
>> > 866.263.9295
>> >
>> > —
>> > NTFSD is sponsored by OSR
>> >
>> > OSR is hiring!! Info at http://www.osr.com/careers
>> >
>> > For our schedule of debugging and file system seminars visit:
>> > http://www.osr.com/seminars
>> >
>> > To unsubscribe, visit the List Server section of OSR Online at
>> > http://www.osronline.com/page.cfm?name=ListServer
>> >
>>
>>
>>
>> —
>> NTFSD is sponsored by OSR
>>
>> OSR is hiring!! Info at http://www.osr.com/careers
>>
>> For our schedule of debugging and file system seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

6

screwguage I might absolutely offtrack here but can your application
be 32 bit? based on my understanding I think you need
MapUserPhysicalPages? I am not sure but I think AWE works in some way
for 32 bit applications on 64 bit platform.

Regards,
Johri