Hello,
In my userland process I need to be able to allocate memory or map file section with page (as opposed to 64K) granularity. After some research I found that there is no easy way of doing that: the closest thing is to use undocumented MEM_DOS_LIM flag for NtMapViewOfSection, which unfortunately is not available on 64-bit.
So the possible solution I can think of is to modify the process’ VAD tree directly.
The questions are: How exactly should I do that? Is there some easier or perhaps better way?
Are there any drawbacks or limitations of not using 64K-align? Is it even possible to implement at all?
Regards
There is, AFAIK, no user level library or system API that guarantees
page-aligned allocation. The usual way we did this was to allocate a
block of memory using new/malloc, that was large enough so that when we
rounded the address up we would get a page-aligned address that spanned an
integral number of pages.
For VirtualAlloc* and memory-mapped file sections, you /must/ use the
allocation granularity, which, this week, in Win32, with the current set
of point releases and hotfixes, is 64K. If you want any other granularity
(and you should use the recommended API to get this value, not just assume
it is 64K), too bad. If you think you have some exceptionally clever way
to accomplish what you want, you are making a huge mistake. Don’t go
there. Live with the allocation granularity that exists. You absolutely,
positively, do not want to reach in and manipulate Data Structures Sacred
To The Kernel unless you know all the proper rituals, such as how to lock
the structure against concurrent access. And vastly more sophisticated
problems. The easiest thing to say is that if you cannot handle file
mappings to granularity boundaries, your design is flawed beyond
redemption. And doing new/malloc to page-aligned boundaries is trivial
arithmetic. Doing VirtualAlloc* to other than allocation-granularity
boundaries is not possible (have you wondered why the name “allocation
boundary” came about?)
You are asking for the impossible with respect to VirtualAlloc* and memory
mapping. Deal with it, because any attempt to “work around” it is
foolish.
As to “drawbacks”: under ideal conditions, you have compromised the
integrity of kernel data structures. You cannot handle concurrent access.
If it works at all, it means that it only works on your machine, using
your processor model and chipset, under the extremely limited conditions
under which you are testing. It will probably fail catastrophically (BSOD
if you are lucky, far worse if you are not) on any other machine or under
actual load conditions. And that’s just off the top of my head,
first-order failure modes.
joe
Hello,
In my userland process I need to be able to allocate memory or map file
section with page (as opposed to 64K) granularity. After some research I
found that there is no easy way of doing that: the closest thing is to use
undocumented MEM_DOS_LIM flag for NtMapViewOfSection, which unfortunately
is not available on 64-bit.
So the possible solution I can think of is to modify the process’ VAD tree
directly.
The questions are: How exactly should I do that? Is there some easier or
perhaps better way?
Are there any drawbacks or limitations of not using 64K-align? Is it even
possible to implement at all?
Regards
NTDEV is sponsored by OSR
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Why do you think that you need to do this?
Directly editing the VAD tree is wholly unsupported. The mechanisms and synchronization required to do that are private to the OS and subject to change from hotfix to hotfix; doing that would be a great way to risk bugchecking all of your customers’ machine on the next automatically installed hotfix.
- S (Msft)
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@mailnull.com
Sent: Monday, May 27, 2013 5:47 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Allocate with page granularity
Hello,
In my userland process I need to be able to allocate memory or map file section with page (as opposed to 64K) granularity. After some research I found that there is no easy way of doing that: the closest thing is to use undocumented MEM_DOS_LIM flag for NtMapViewOfSection, which unfortunately is not available on 64-bit.
So the possible solution I can think of is to modify the process’ VAD tree directly.
The questions are: How exactly should I do that? Is there some easier or perhaps better way?
Are there any drawbacks or limitations of not using 64K-align? Is it even possible to implement at all?
Regards
NTDEV is sponsored by OSR
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
> In my userland process I need to be able to allocate memory or map file section with page (as
opposed to 64K) granularity
Just use 64K granularity and you’re fine.
Also, you can reserve with 64K granularity and then IIRC commit with page granularity.
–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com
Hello,
Thanks for your replies.
Yes, I am fully aware that what I want to do is not supported in any way. But this is more like a personal/research project, so I don’t really need to worry about any customers; of course, otherwise I would’ve never considered doing something like that.
As far as I know, allocating/mapping with page granularity is possible after all, and the kernel uses it for things like PEB or TEB. The existance of MEM_DOS_LIM flag also suggests that it is possible (didn’t try it, but I think it should work).
The reason for what I’m trying to do is that I need to mix both memory allocations and file mappings in one 64K segment. Better solution to do that would be nice.
As for synchronization/concurrent access (for editing VAD directly), I think that it will be enough to use AddressCreationLock mutex from internal EPROCESS structure (opaque & unsupported, I know 
)
Regards
You do realize this flag is not only undocumented, it is not defined in
the Microsoft supplied headers. So you are basing something on data from
the ReactOS which many times get the Microsoft kernel wrong (or at least
it did when I had source access to the Windows kernel). You won’t even
get the hint that Microsoft has changed things which people get when
they use undocumented but defined by Microsoft flags (since the flags go
away on a change).
All I can say is please tell us the company name and the product name so
we can boycott this piece of shirt when it comes out. Step back as has
been suggested and accept the 64KB limit and break that into pages
yourself, or expect something that will not work.
Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
“xxxxx@mailnull.com”
wrote in message news:xxxxx@ntdev:
> Hello,
>
> Thanks for your replies.
> Yes, I am fully aware that what I want to do is not supported in any way. But this is more like a personal/research project, so I don’t really need to worry about any customers; of course, otherwise I would’ve never considered doing something like that.
>
> As far as I know, allocating/mapping with page granularity is possible after all, and the kernel uses it for things like PEB or TEB. The existance of MEM_DOS_LIM flag also suggests that it is possible (didn’t try it, but I think it should work).
> The reason for what I’m trying to do is that I need to mix both memory allocations and file mappings in one 64K segment. Better solution to do that would be nice.
> As for synchronization/concurrent access (for editing VAD directly), I think that it will be enough to use AddressCreationLock mutex from internal EPROCESS structure (opaque & unsupported, I know )
>
> Regards
> All I can say is please tell us the company name and the product name so we can boycott this
piece of shirt when it comes out.
Don, you are just incredible - the OP has stated VERY clearly that he is doing his hobby research project.
However, you still go on with your mantra. Aren’t you bored with repeating the same thing over and over again, and doing it for YEARS???
Anton Bassov
> As far as I know, allocating/mapping with page granularity is possible after all, and the kernel uses it
for things like PEB or TEB.
IIRC, ZwAllocateVirtualMemory() takes 'ZeroBits" parameter that specifies the number of high-order bits in the
target address. According to the Nebbett’s book, it has to be less than 21 (for32-bit system, of course).
This strongly suggests that allocations on page granularity are, indeed, possible. For the fun of doing it,
you may try it…
Anton Bassov
> Hello,
Thanks for your replies.
Yes, I am fully aware that what I want to do is not supported in any way.
But this is more like a personal/research project, so I don’t really need
to worry about any customers; of course, otherwise I would’ve never
considered doing something like that.As far as I know, allocating/mapping with page granularity is possible
after all, and the kernel uses it for things like PEB or TEB. The
existance of MEM_DOS_LIM flag also suggests that it is possible (didn’t
try it, but I think it should work).
The reason for what I’m trying to do is that I need to mix both memory
allocations and file mappings in one 64K segment. Better solution to do
that would be nice.
I don’t even see how this could make sense in general. In the localized
case, where you need to allocate object space in the mapped file, you have
to write an allocator using based pointers, which can be relatively
simple, but there is no way you can get any of the standard allocators to
direct heap allocations to a specific address, and there is no reason to
have the general heap share space with mapped pages.
You are attempting to implement something which is unsupported, and your
description of why you need to do this is in no way convincing. Whatever
you are doing, your design is out-of-touch with reality.
I presume you are doing this development in a sacrificial VM, or on a
dedicated machine whose disk contents you are prepared to lose entirely
during any debug session during your development, or after you start to
use the modificatios you make.
Seriously, why do you think this idea could make sense?
joe
As for synchronization/concurrent access (for editing VAD directly), I
think that it will be enough to use AddressCreationLock mutex from
internal EPROCESS structure (opaque & unsupported, I knowRegards
NTDEV is sponsored by OSR
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
>> As far as I know, allocating/mapping with page granularity is possible
> after all, and the kernel uses it
>for things like PEB or TEB.IIRC, ZwAllocateVirtualMemory() takes 'ZeroBits" parameter that specifies
the number of high-order bits in the
target address. According to the Nebbett’s book, it has to be less than 21
(for32-bit system, of course).
This strongly suggests that allocations on page granularity are, indeed,
possible. For the fun of doing it,
you may try it…
But remember that the higher-level VirtualAlloc has different restrictions
based on the input flags: for reservation, it must be the allocation
granularity (this week, on x86 platforms, 64K) and for commit it can be
page granularity (this week, on x86 platforms, 4K). What addresses it
will accept will depend on the nature of the operation, and specifying a
4K boundary that is not a 64K boundary does not mean it will honor the
request; more likely it will fail and the error code will be “invalid
parameter”.
Don’s mantra is well-founded; people who write “hobby code” like this may
actually succeed in their very limited environment, and go on to think
such solutions should appear in products.
There is no point to doing a “hobby” project that makes ZERO sense; there
is nothing to be learned by corrupting the internals of an operating
system, particularly when the goal seems so bizarre: to be able to mix
file mappings and heap allocation in the same 64K section! I fail to
understand what the ability to do this adds in any way to the application
environment. Since there is no way to direct a storage allocator to a
specific address, and no reason to need to play in the remaining 64K by
using general allocation, this has ZERO benefit in any application. If
the reasoning is “otherwise, I’m wasting up to 60K”, that reasoning is
completely silly. 60K is less than 0.003% of the default 2GB address
space. Or, to put it another way, you need close to 400
concurrently-mapped files before you are wasting 1% of the address space.
The OP gave no rationale as to why this project would be justifiable, and
a “hobby” project that involves mucking with undocumented data structures
without any locking, or even a clue as to how they are created, managed,
and released by the kernel does not “teach” anything. I will avoid using
the l-word here, but I suspect that doing something like this in a
fully-open-source OS, violating analogous restrictions, would not teach
anything valuable, either. And I say this as someone who has spent his
life teaching, and pushing software to its limits and beyond. A teaching
experience has to actually /teach/ something consistent with the effort.
Here, there is unending effort for zero payback, which is about as bad as
it can get. As a responsible teacher, I would never assign a task like
this to a student. I would assign the student a task that required
learning something useful.
joe
Anton Bassov
NTDEV is sponsored by OSR
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Anton,
If you believe a claim of hobby programming or that if it is hobby
programming someone won’t try to turn it into a commercial product, I
have a bridge or two to sell you. Half the crap on CodeProject is
“hobby programming” I stand by my statement.
Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
“xxxxx@hotmail.com” wrote in message
news:xxxxx@ntdev:
> > All I can say is please tell us the company name and the product name so we can boycott this
> > piece of shirt when it comes out.
>
>
> Don, you are just incredible - the OP has stated VERY clearly that he is doing his hobby research project.
> However, you still go on with your mantra. Aren’t you bored with repeating the same thing over and over again, and doing it for YEARS???
>
>
> Anton Bassov
xxxxx@flounder.com wrote:
There is no point to doing a “hobby” project that makes ZERO sense
The OP may be looking for ways to run Linux binaries inside a Win32 process.
I read somewhere that Windows’ 64K allocation granularity makes it difficult
to emulate the Linux user-mode address space layout.
> xxxxx@flounder.com wrote:
> There is no point to doing a “hobby” project that makes ZERO sense
>
> The OP may be looking for ways to run Linux binaries inside a Win32
> process.
> I read somewhere that Windows’ 64K allocation granularity makes it
> difficult
> to emulate the Linux user-mode address space layout.
>
>
But that is not the reason the OP gives. And the abstraction of memory
allocation should isolate the app programmer from such differences.
joe
NTDEV is sponsored by OSR
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
xxxxx@flounder.com wrote:
But that is not the reason the OP gives.
I did label my speculation as a “wild guess”.
“What did the OP *really* want?” is a classic NTDEV parlor game 
And the abstraction of memory allocation should
> isolate the app programmer from such differences.
Of course “normal” application code that uses malloc and free won’t care.
The problem would be “unusual” code that uses mmap and makes assumptions
about how it works.
Also don’t forget that the existence of large pages negates the idea that
page size is a constant.
wrote in message news:xxxxx@ntdev…
> As far as I know, allocating/mapping with page granularity is possible
> after all, and the kernel uses it
>for things like PEB or TEB.IIRC, ZwAllocateVirtualMemory() takes 'ZeroBits" parameter that specifies
the number of high-order bits in the
target address. According to the Nebbett’s book, it has to be less than 21
(for32-bit system, of course).
This strongly suggests that allocations on page granularity are, indeed,
possible. For the fun of doing it,
you may try it…
But remember that the higher-level VirtualAlloc has different restrictions
based on the input flags: for reservation, it must be the allocation
granularity (this week, on x86 platforms, 64K) and for commit it can be
page granularity (this week, on x86 platforms, 4K). What addresses it
will accept will depend on the nature of the operation, and specifying a
4K boundary that is not a 64K boundary does not mean it will honor the
request; more likely it will fail and the error code will be “invalid
parameter”.
Don’s mantra is well-founded; people who write “hobby code” like this may
actually succeed in their very limited environment, and go on to think
such solutions should appear in products.
There is no point to doing a “hobby” project that makes ZERO sense; there
is nothing to be learned by corrupting the internals of an operating
system, particularly when the goal seems so bizarre: to be able to mix
file mappings and heap allocation in the same 64K section! I fail to
understand what the ability to do this adds in any way to the application
environment. Since there is no way to direct a storage allocator to a
specific address, and no reason to need to play in the remaining 64K by
using general allocation, this has ZERO benefit in any application. If
the reasoning is “otherwise, I’m wasting up to 60K”, that reasoning is
completely silly. 60K is less than 0.003% of the default 2GB address
space. Or, to put it another way, you need close to 400
concurrently-mapped files before you are wasting 1% of the address space.
The OP gave no rationale as to why this project would be justifiable, and
a “hobby” project that involves mucking with undocumented data structures
without any locking, or even a clue as to how they are created, managed,
and released by the kernel does not “teach” anything. I will avoid using
the l-word here, but I suspect that doing something like this in a
fully-open-source OS, violating analogous restrictions, would not teach
anything valuable, either. And I say this as someone who has spent his
life teaching, and pushing software to its limits and beyond. A teaching
experience has to actually /teach/ something consistent with the effort.
Here, there is unending effort for zero payback, which is about as bad as
it can get. As a responsible teacher, I would never assign a task like
this to a student. I would assign the student a task that required
learning something useful.
joe
Anton Bassov
NTDEV is sponsored by OSR
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
What??? You mean all pages on all platforms are 4096 bytes??? (Remember
the late and unlamented DEC Alpha, with 8K pages).
And you mean not every operating system can do a bitwise emulation of sone
other operating system?
Wow, I learn something new every day! (such as that it appears some people
would believe ideas like this)
joe
Also don’t forget that the existence of large pages negates the idea
that
page size is a constant.wrote in message news:xxxxx@ntdev…
>> As far as I know, allocating/mapping with page granularity is possible
>> after all, and the kernel uses it
>>for things like PEB or TEB.
>
> IIRC, ZwAllocateVirtualMemory() takes 'ZeroBits" parameter that
> specifies
> the number of high-order bits in the
> target address. According to the Nebbett’s book, it has to be less than
> 21
> (for32-bit system, of course).
> This strongly suggests that allocations on page granularity are, indeed,
> possible. For the fun of doing it,
> you may try it…
>But remember that the higher-level VirtualAlloc has different restrictions
based on the input flags: for reservation, it must be the allocation
granularity (this week, on x86 platforms, 64K) and for commit it can be
page granularity (this week, on x86 platforms, 4K). What addresses it
will accept will depend on the nature of the operation, and specifying a
4K boundary that is not a 64K boundary does not mean it will honor the
request; more likely it will fail and the error code will be “invalid
parameter”.Don’s mantra is well-founded; people who write “hobby code” like this may
actually succeed in their very limited environment, and go on to think
such solutions should appear in products.There is no point to doing a “hobby” project that makes ZERO sense; there
is nothing to be learned by corrupting the internals of an operating
system, particularly when the goal seems so bizarre: to be able to mix
file mappings and heap allocation in the same 64K section! I fail to
understand what the ability to do this adds in any way to the application
environment. Since there is no way to direct a storage allocator to a
specific address, and no reason to need to play in the remaining 64K by
using general allocation, this has ZERO benefit in any application. If
the reasoning is “otherwise, I’m wasting up to 60K”, that reasoning is
completely silly. 60K is less than 0.003% of the default 2GB address
space. Or, to put it another way, you need close to 400
concurrently-mapped files before you are wasting 1% of the address space.
The OP gave no rationale as to why this project would be justifiable, and
a “hobby” project that involves mucking with undocumented data structures
without any locking, or even a clue as to how they are created, managed,
and released by the kernel does not “teach” anything. I will avoid using
the l-word here, but I suspect that doing something like this in a
fully-open-source OS, violating analogous restrictions, would not teach
anything valuable, either. And I say this as someone who has spent his
life teaching, and pushing software to its limits and beyond. A teaching
experience has to actually /teach/ something consistent with the effort.
Here, there is unending effort for zero payback, which is about as bad as
it can get. As a responsible teacher, I would never assign a task like
this to a student. I would assign the student a task that required
learning something useful.
joe
>
> Anton Bassov
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
NTDEV is sponsored by OSR
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer