Hi gurus,
Do you know whether windows kernel tries to use large pages for page/non-paged pool allocations (Assuming the allocations are big enough and aligned). The docs state that it does so for the " initial part of nonpaged pool " and MmIoSpace requests, but nothing else.
If windows doesn’t try to do that, is there some way to allocate virtual kernel space that is mapped by large pages?
Thanks,
Eran.
Surely this is the undocumented implementation detail and you should not
rely on it, but at least older Windows like NT4 created the hard-wired
immutable global map of [0…512MB) physical to [0x80000000…0xa0000000)
kernel-only virtual, this mapping was established using large pages at boot.
This mapping helped MmMapLockedPages(KernelMode) a lot for 1 page MDLs if
the physical address is < 512MB - just add 0x80000000 to the physical address
and all is done, and unmap is a no-op.
Also, this mapping was used for the so-called “primary” nonpaged pool.
I expect the feature to be present in modern Windows too.
As about dynamic allocation of large pages in the kernel - don’t know.
–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
wrote in message news:xxxxx@ntdev…
> Hi gurus,
> Do you know whether windows kernel tries to use large pages for
page/non-paged pool allocations (Assuming the allocations are big enough and
aligned). The docs state that it does so for the " initial part of nonpaged
pool " and MmIoSpace requests, but nothing else.
> If windows doesn’t try to do that, is there some way to allocate virtual
kernel space that is mapped by large pages?
>
> Thanks,
> Eran.
>
I don’t know the answer to this beyond Maxim already said.
For the sake of argument, if you really wanted to do this, I think you
could have a user process allocate the memory for you with
VirtualAlloc/VirtualAllocEx, specifying MEM_LARGE_PAGES.
http://msdn2.microsoft.com/en-us/library/aa366887(VS.85).aspx
I’ve never tried this, there are a number of limitations, most notablt
that this isn’t an option on XP, and implications as well, most notably
that the memory allocated is not part of the working set.
I’m not sure that this is best of ideas in practice, but if you’re
really interested in this, I would disassemble VirtualAlloc() with
MEM_LARGE_PAGES and see what’s what.
Good luck,
mm
Maxim S. Shatskih wrote:
Surely this is the undocumented implementation detail and you should not
rely on it, but at least older Windows like NT4 created the hard-wired
immutable global map of [0…512MB) physical to [0x80000000…0xa0000000)
kernel-only virtual, this mapping was established using large pages at boot.This mapping helped MmMapLockedPages(KernelMode) a lot for 1 page MDLs if
the physical address is < 512MB - just add 0x80000000 to the physical address
and all is done, and unmap is a no-op.Also, this mapping was used for the so-called “primary” nonpaged pool.
I expect the feature to be present in modern Windows too.
As about dynamic allocation of large pages in the kernel - don’t know.
xxxxx@topio.com wrote:
Do you know whether windows kernel tries to use large pages for page/non-paged pool allocations (Assuming the allocations are big enough and aligned). The docs state that it does so for the " initial part of nonpaged pool " and MmIoSpace requests, but nothing else.
If windows doesn’t try to do that, is there some way to allocate virtual kernel space that is mapped by large pages?
Why do you care about this?
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
> If windows doesn’t try to do that, is there some way to allocate virtual
kernel space that is mapped by large pages?
ZwMapViewOfSection with MEM_LARGE_PAGES flag. I haven’t tried it but
VirtualAlloc relies on ZwMapViewOfSection so ZwMapViewOfSection should
support this flag. Also there is a post from Jeff McCashland from Microsoft
that claims it too.
Keep in mind that these large pages are nonpageable.
Dmitriy Budko
VMware
> ZwMapViewOfSection with MEM_LARGE_PAGES flag. I haven’t tried it but VirtualAlloc relies on
ZwMapViewOfSection so ZwMapViewOfSection should support this flag.
A couple of observations:
VirtualAlloc() relies upon ZwAllocateVirtualMemory(), rather than upon ZwMapViewOfSection(). The former one is not exported by ntoskrnl.exe. Therefore, there is no guarantee that MEM_LARGE_PAGES flag is usable in the KM
MEM_LARGE_PAGES flag is supported only on OS versions above XP
Anton Bassov
> > ZwMapViewOfSection with MEM_LARGE_PAGES flag. I haven’t tried it but
VirtualAlloc relies on
> ZwMapViewOfSection so ZwMapViewOfSection should support this flag.A couple of observations:
- VirtualAlloc() relies upon ZwAllocateVirtualMemory(), rather than upon
ZwMapViewOfSection(). The former one is not exported by ntoskrnl.exe.
Therefore, there is no guarantee that MEM_LARGE_PAGES flag is usable in
the KM
Anton Bassov
Anton,
you are right: VirtualAlloc() relies upon ZwAllocateVirtualMemory but
ZwAllocateVirtualMemory is exported from kernel and is also has the
prototype in <ntifs.h> and is documented:
http://msdn2.microsoft.com/en-us/library/ms800949.aspx
Anyway, I am pretty sure that the all modern DDIs (and majority of Win32 APIs
like CreateFileMapping) that accept standard MEM_ flags will also accept
MEM_LARGE_PAGES / SEC_LARGE_PAGES.
Dmitriy Budko
VMware</ntifs.h>
But it only allocates user-mode pages, not kernel-mode pages. You can allocate the pages in the system process if you want to keep them out of the grubby little hands of an application, but they won’t be visible in all contexts.
Unless you map them into system VA space, at which point you’re back to not using large pages.
I think (to me) this all goes back to the question of why the OP wanted to force this to happen.
-p
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Dmitriy Budko
Sent: Monday, February 25, 2008 4:41 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Large pages allocation
> ZwMapViewOfSection with MEM_LARGE_PAGES flag. I haven’t tried it but
VirtualAlloc relies on
> ZwMapViewOfSection so ZwMapViewOfSection should support this flag.A couple of observations:
- VirtualAlloc() relies upon ZwAllocateVirtualMemory(), rather than upon
ZwMapViewOfSection(). The former one is not exported by ntoskrnl.exe.
Therefore, there is no guarantee that MEM_LARGE_PAGES flag is usable in
the KM
Anton Bassov
Anton,
you are right: VirtualAlloc() relies upon ZwAllocateVirtualMemory but
ZwAllocateVirtualMemory is exported from kernel and is also has the
prototype in <ntifs.h> and is documented:
http://msdn2.microsoft.com/en-us/library/ms800949.aspx
Anyway, I am pretty sure that the all modern DDIs (and majority of Win32 APIs
like CreateFileMapping) that accept standard MEM_ flags will also accept
MEM_LARGE_PAGES / SEC_LARGE_PAGES.
Dmitriy Budko
VMware
—
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</ntifs.h>
Dmitry,
ZwAllocateVirtualMemory is exported from kernel
Indeed, it is exported from the kernel, but it is exported via SSDT, rather than PE header, which means kernel-mode components are not supposed to call it. Therefore, it just cannot be prototyped in ifs.h. Certainly you can call it simply by its address, but be ready for “surprises” if you do it this way. ZwMapViewOfSection() is a totally different beast - it is exported by the kernel via PE header in both Nt… and Zw… forms , which means it is supposed to be called from the kernel mode in either form (the latter one is even documented in WDK). Therefore,
it may “misunderstand” MEM_LARGE_PAGES flag - after all, once ZwAllocateVirtualMemory()
is meant to be called only by the user-mode components, all allocations that it does are meant to be specific to the target process, rather than having system-wide implications…
Anyway, I am pretty sure that the all modern DDIs (and majority of Win32 APIs
like CreateFileMapping) that accept standard MEM_ flags will also
accept MEM_LARGE_PAGES / SEC_LARGE_PAGES.
Well, if kernel does not expect it on OS versions <=XP, this flag, apparently, just does not
have any meaning to the older versions of kernel…
Anton Bassov
Unless I’m missing something, it’s exported by the PE header, and there
is also a linker member entry for it in the export library, at least on
XP SP2:
Dump of file c:\windows\system32\ntoskrnl.exe
File Type: EXECUTABLE IMAGE
Section contains the following exports for ntoskrnl.exe
00000000 characteristics
42250F77 time date stamp Tue Mar 01 19:57:27 2005
0.00 version
1 ordinal base
1483 number of functions
1483 number of names
ordinal hint RVA name
…
1283 502 0000C3D6 ZwAllocateVirtualMemory
…
Microsoft (R) COFF/PE Dumper Version 8.00.50727.42
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file ntoskrnl.lib
File Type: LIBRARY
Archive member name at 8: /
472C29EF time/date Sat Nov 03 03:57:35 2007
uid
gid
0 mode
1F1D9 size
correct header end
3937 public symbols
45D58 _ZwAllocateVirtualMemory@24
45D58 __imp__ZwAllocateVirtualMemory@24
There’s also a linker member entry for WLH in the 6001.18000 WDK, but I
don’t currently have a target that is running S2008 to check for to see
whether it is exported or not.
mm
xxxxx@hotmail.com wrote:
Dmitry,
> ZwAllocateVirtualMemory is exported from kernel
Indeed, it is exported from the kernel, but it is exported via SSDT, rather than PE header, which means kernel-mode components are not supposed to call it. Therefore, it just cannot be prototyped in ifs.h. Certainly you can call it simply by its address, but be ready for “surprises” if you do it this way. ZwMapViewOfSection() is a totally different beast - it is exported by the kernel via PE header in both Nt… and Zw… forms , which means it is supposed to be called from the kernel mode in either form (the latter one is even documented in WDK). Therefore,
it may “misunderstand” MEM_LARGE_PAGES flag - after all, once ZwAllocateVirtualMemory()
is meant to be called only by the user-mode components, all allocations that it does are meant to be specific to the target process, rather than having system-wide implications…> Anyway, I am pretty sure that the all modern DDIs (and majority of Win32 APIs
> like CreateFileMapping) that accept standard MEM_ flags will also
> accept MEM_LARGE_PAGES / SEC_LARGE_PAGES.Well, if kernel does not expect it on OS versions <=XP, this flag, apparently, just does not
have any meaning to the older versions of kernel…Anton Bassov
Martin,
It looks like I am spreading misinformation all over the place - I have a weird feeling that I’ve never seen ZwAllocateVirtualMemory() in ntoskrnl.exe’s dump (unfortunately I haven’t got my Windows machine in front of me so that I cannot immediately double-check it). In any case, according to header dump that you have provided I am wrong, so that it already does not make sense to double- check it - we already have an answer …
Anton Bassov
I hardly ever use any of the km Virtual() functions, and I can never
remember what is and what isn’t exported either. The lack of and export
for ZwProtectVirtualMemory is what sticks in my head, because it’s the
only one I really ever use, but I just can’t seem to remember that well
enough to feel confident on the rare occasions that I use try to use it.
It all just gets jumbled, and when I can’t find
ZwProtectVirtualMemory, then I start looking for things like
NtProtectVirtualMemory, ZwVirtualProtect, Mm… The net result for me
is that every time anything about km Virtual() comes up, I end up going
down the same road everytime, and generally I just avoid the issue if I
can. The only reason I looked tonight is that I was curious to see what
the deal was, as both you and Dmitriy most certainly know your stuff.
Cheers,
mm
xxxxx@hotmail.com wrote:
Martin,
It looks like I am spreading misinformation all over the place - I have a weird feeling that I’ve never seen ZwAllocateVirtualMemory() in ntoskrnl.exe’s dump (unfortunately I haven’t got my Windows machine in front of me so that I cannot immediately double-check it). In any case, according to header dump that you have provided I am wrong, so that it already does not make sense to double- check it - we already have an answer …
Anton Bassov
> Unless you map them into system VA space, at which point you’re back to
not using large pages.
I think (to me) this all goes back to the question of why the OP wanted to
force this to happen.
One can directly modify system PTEs after that to use already allocated
physical large pages. IMHO, it would work but it would be a nasty hack.
Yes, the answer depends on the usage of large pages.
Dmitriy Budko
VMware
Dmitriy Budko wrote:
One can directly modify system PTEs after that to use already
allocated
physical large pages. IMHO, it would work but it would be a nasty
hack.
You mean in PDE right? PSE will need to be enabled in CR4 and PS set in PDE.
Also what linear address to map it to, unless it’s a PS PDE already you could loose link to a PT…
Yes, the answer depends on the usage of large pages.
Dmitriy Budko
VMware
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Hi all,
Thanks for all your responses.
My use case is the following:
My kernel driver allocates big chunk of non-paged memory (In 64 bits machines it might be 1 GB).
This memory is also shared with a user-mode process (Yes I know it is not recommended, but my requirements are that the memory remains available even when the process is down). Now, data flows to the big memory pool via DMA from an HBA that acts as a target device. Once data is in the pool, it is processed first in kernel mode, and then in user-mode. The HBA does supports scatter/gather, so virtual memory is fine. However, in order to scale, I would like to reduce the amount of scatter/gather list elements in my DMA transfer. This is why I thought about large pages as those seems to fit me exactly right.
Thanks,
Eran.
“Maxim S. Shatskih” wrote:
Surely this is the undocumented implementation detail and you should
not
rely on it, but at least older Windows like NT4 created the hard-wired
immutable global map of [0…512MB) physical to [0x80000000…0xa0000000)
kernel-only virtual, this mapping was established using large pages at
boot.This mapping helped MmMapLockedPages(KernelMode) a lot for 1 page MDLs
if
the physical address is < 512MB - just add 0x80000000 to the physical
address
and all is done, and unmap is a no-op.Also, this mapping was used for the so-called “primary” nonpaged pool.
I expect the feature to be present in modern Windows too.
Only NT4 and Win2K worked like this. The problem with this scheme
is that entire memory below 512 MB was mapped as write-back
cacheable. So applications were not able to use different cache attributes
(e.g. PAGE_NOCACHE) because doing this would create conflicting
TLB entries for the same physical address, which is not allowed.
–
This posting is provided “AS IS” with no warranties, and confers no
rights.