Memory Mapped IO

NTDEV
1

I have just noticed that when a process starts and loads all the necessary
dll’s required for the process, the virtual address space in user-mode
between 0x10000 - 07FFFFFFF is fragmented and in some case can even cause
problems when you require to allocate a large chunk of contiguous memory.
I don’t want to rebase all the system dll’s unless there is a windows system
option or boot option which can force all dll’s to load at the top end of
memory.

<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office"
/\> I have a requirement where I need to be able to process extremely large data files which can be in excess of 20GB's however I need to be able to map at least 512 Megabytes of the file into memory. Obviously I need to unmap and remap 512MB at various positions within the file. To do this I need to guarantee that I can remap 512MB once I have unmapped it, without other threads in the process fragmenting the previous mapped virtual address space! I thought I could achieve this by performing LPVOID pBaseAddress = VirtualAlloc(NULL, 0x20000000 MEM_RESERVE PAGE_READWRITE) Then passing the base address to MapViewOfFileEx but I cant seem to get it to work. Is there a way to guarantee that 512MB can be mapped once the previous section has been unmapped? Ideally I would like to reserve a chunk of memory for mapping my file then the process initially starts up. General Dynamics United Kingdom Limited Registered in England and Wales No. 1911653 Registered Office: 100 New Bridge Street, London, EC4V 6JA
2

Why are u trying to allocate a gigantic chunk of memory if you want to map a
view of your file?

Unless you dont really want direct access open the file and use
CreateFileMapping.

I think this should explain enough for you.
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base
/creating_a_file_mapping_object.asp

Regards,

Rob Linegar
Software Engineer
Data Encryption Systems Limited
www.des.co.uk | www.deslock.co.uk

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of James Dunning
Sent: 10 June 2004 13:37
To: Windows System Software Devs Interest List
Subject: [ntdev] Memory Mapped IO

I have just noticed that when a process starts and loads all the necessary
dll’s required for the process, the virtual address space in user-mode
between 0x10000 - 07FFFFFFF is fragmented and in some case can even cause
problems when you require to allocate a large chunk of contiguous memory.
I don’t want to rebase all the system dll’s unless there is a windows system
option or boot option which can force all dll’s to load at the top end of
memory.

I have a requirement where I need to be able to process extremely large data
files which can be in excess of 20GB’s however I need to be able to map at
least 512 Megabytes of the file into memory. Obviously I need to unmap and
remap 512MB at various positions within the file.

To do this I need to guarantee that I can remap 512MB once I have unmapped
it, without other threads in the process fragmenting the previous mapped
virtual address space!

I thought I could achieve this by performing

LPVOID pBaseAddress = VirtualAlloc(NULL, 0x20000000 MEM_RESERVE
PAGE_READWRITE)

Then passing the base address to MapViewOfFileEx but I cant seem to get it
to work.

Is there a way to guarantee that 512MB can be mapped once the previous
section has been unmapped? Ideally I would like to reserve a chunk of memory
for mapping my file then the process initially starts up.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@des.co.uk
To unsubscribe send a blank email to xxxxx@lists.osr.com

General Dynamics United Kingdom Limited
Registered in England and Wales No. 1911653
Registered Office: 100 New Bridge Street, London, EC4V 6JA

3

Silly suggestion time: How about using an Athlon64/Opteron system and
Windows XP for 64-bit extended systems, where you can EASILY have more than
4GB of address space. In fact, even a 32-bit app will work just fine there,
because it’s given almost all of the 4GB of address space to play with in
user mode, only a tiny little bit is reserved by the kernel.

Another option might be to try the /3GB switch on the boot.ini line, and
compile your app to “know about large memory” [pass /LARGEADDRESSAWARE to
the linker]. Then you have another 1GB to play with.

I’m sure lots of other list members will come up with less silly
suggestions, but you never know…

–
Mats

-----Original Message-----
From: James Dunning [mailto:xxxxx@generaldynamics.uk.com]
Sent: Thursday, June 10, 2004 1:37 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Memory Mapped IO

I have just noticed that when a process starts and loads all the necessary
dll’s required for the process, the virtual address space in user-mode
between 0x10000 - 07FFFFFFF is fragmented and in some case can even cause
problems when you require to allocate a large chunk of contiguous memory.
I don’t want to rebase all the system dll’s unless there is a windows system
option or boot option which can force all dll’s to load at the top end of
memory.

I have a requirement where I need to be able to process extremely large data
files which can be in excess of 20GB’s however I need to be able to map at
least 512 Megabytes of the file into memory. Obviously I need to unmap and
remap 512MB at various positions within the file.

To do this I need to guarantee that I can remap 512MB once I have unmapped
it, without other threads in the process fragmenting the previous mapped
virtual address space!

I thought I could achieve this by performing

LPVOID pBaseAddress = VirtualAlloc(NULL, 0x20000000 MEM_RESERVE
PAGE_READWRITE)

Then passing the base address to MapViewOfFileEx but I cant seem to get it
to work.

Is there a way to guarantee that 512MB can be mapped once the previous
section has been unmapped? Ideally I would like to reserve a chunk of memory
for mapping my file then the process initially starts up.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@3dlabs.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

General Dynamics United Kingdom Limited

Registered in England and Wales No. 1911653

Registered Office: 100 New Bridge Street, London, EC4V 6JA

4

lol, i knew someone would come up with the 64bit processor suggestion :wink: i
only wish i could go 64bit as it would be the answer to all my problems.
unfortunately due to the strict security requirements this is a route i can
not take, sadly.

I do know how to use memory mapped files, but i just think Its just really
annoying that i cant use utilise the MapViewOfFileEx and specify the base
address that i can reused time and time again which i know has been
allocated and reserved…

basically i am dealing with large streams of compressed imagery of 20GB’s
which needs to be decompressed, the decompressed data is currently stored to
a 3GB cyclic buffer using a memory mapped file… and this file lives on an
extremely fast raid with transfer speeds of 400MB/s.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of
xxxxx@3Dlabs.com
Sent: 10 June 2004 14:11
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Memory Mapped IO

Silly suggestion time: How about using an Athlon64/Opteron system and
Windows XP for 64-bit extended systems, where you can EASILY have more than
4GB of address space. In fact, even a 32-bit app will work just fine there,
because it’s given almost all of the 4GB of address space to play with in
user mode, only a tiny little bit is reserved by the kernel.

Another option might be to try the /3GB switch on the boot.ini line, and
compile your app to “know about large memory” [pass /LARGEADDRESSAWARE to
the linker]. Then you have another 1GB to play with.

I’m sure lots of other list members will come up with less silly
suggestions, but you never know…

–
Mats

-----Original Message-----
From: James Dunning [mailto:xxxxx@generaldynamics.uk.com]
Sent: Thursday, June 10, 2004 1:37 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Memory Mapped IO

I have just noticed that when a process starts and loads all the necessary
dll’s required for the process, the virtual address space in user-mode
between 0x10000 - 07FFFFFFF is fragmented and in some case can even cause
problems when you require to allocate a large chunk of contiguous memory.
I don’t want to rebase all the system dll’s unless there is a windows system
option or boot option which can force all dll’s to load at the top end of
memory.

I have a requirement where I need to be able to process extremely large data
files which can be in excess of 20GB’s however I need to be able to map at
least 512 Megabytes of the file into memory. Obviously I need to unmap and
remap 512MB at various positions within the file.

To do this I need to guarantee that I can remap 512MB once I have unmapped
it, without other threads in the process fragmenting the previous mapped
virtual address space!

I thought I could achieve this by performing

LPVOID pBaseAddress = VirtualAlloc(NULL, 0x20000000 MEM_RESERVE
PAGE_READWRITE)

Then passing the base address to MapViewOfFileEx but I cant seem to get it
to work.

Is there a way to guarantee that 512MB can be mapped once the previous
section has been unmapped? Ideally I would like to reserve a chunk of memory
for mapping my file then the process initially starts up.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@3dlabs.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

General Dynamics United Kingdom Limited

Registered in England and Wales No. 1911653

Registered Office: 100 New Bridge Street, London, EC4V 6JA

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@generaldynamics.uk.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

General Dynamics United Kingdom Limited
Registered in England and Wales No. 1911653
Registered Office: 100 New Bridge Street, London, EC4V 6JA

5

On Thu, 10 Jun 2004 13:37:17 +0100, James Dunning
wrote:

> I have just noticed that when a process starts and loads all the
> necessary
> dll’s required for the process, the virtual address space in user-mode
> between 0x10000 - 07FFFFFFF is fragmented and in some case can even cause
> problems when you require to allocate a large chunk of contiguous memory.
> I don’t want to rebase all the system dll’s unless there is a windows
> system
> option or boot option which can force all dll’s to load at the top end of
> memory.
>
>
>
> I have a requirement where I need to be able to process extremely large
> data
> files which can be in excess of 20GB’s however I need to be able to map
> at
> least 512 Megabytes of the file into memory. Obviously I need to unmap
> and
> remap 512MB at various positions within the file.
>
>
> To do this I need to guarantee that I can remap 512MB once I have
> unmapped
> it, without other threads in the process fragmenting the previous mapped
> virtual address space!
>
>
> I thought I could achieve this by performing
>
>
> LPVOID pBaseAddress = VirtualAlloc(NULL, 0x20000000 MEM_RESERVE
> PAGE_READWRITE)
>
>
> Then passing the base address to MapViewOfFileEx but I cant seem to get
> it
> to work.
>

Its a small but simple mistake: Until you call VirtualFree to unreserve
that hunk of Virtual Address Space, Your own reservation will block
MapViewOfFileEx from using it (If it didn’t, DLLs could get loaded
there by mistake).

Possible solutions:

a) If you always have at least one 512MB hunk mapped, just delay your
call to UnMapViewOfFile to just before the MapViewOfFile call.

b) If there are periods where no 512MB hunk is mapped, do use VirtualAlloc
as suggested, but call VirtualFree (MEM_RELEASE) just before
MapViewOfFileEx
and VirtualAlloc(MEM_RESERVE) just after UnMapViewOfFile.

In either case, passing a specific base address to these calls is only
necessary if your code would fail at any other address. Otherwise leaving
that out increases your chance of success by allowing you to succeed if
something eats the head or tail of your allocation while still leaving
enough free space at the other end to allow it to go to a slightly
different
base address.

Also in either case make sure other threads in your own code isn’t trying
to
load dlls or allocate memory at the very moment of doing the free/reserve
pair of calls.

>
> Is there a way to guarantee that 512MB can be mapped once the previous
> section has been unmapped? Ideally I would like to reserve a chunk of
> memory
> for mapping my file then the process initially starts up.
>

In my experience the two biggest thieves of virtual addresses in the lowest
GB of process space are:
1. The stacks and heaps. If you do the reserve thing early enough
(Such as DllMain of a statically referenced Dll), there should be
just one stack and less than 3 heaps.

2. “Load in all processes” dlls that come with all sorts of
miscellaneous
utilities: Firewalls, Antivirus, Display Driver “extras”, East Asian
keyboard drivers from Microsoft, Advanced Mouse and Keyboard drivers
etc. Fortunately, many of those don’t load until at at least some of
the DllMain functions have been run.

Jakob

P.S.

A little plea to those here who put “load in all processes” dlls in their
drivers and utilities: Please try to set your base addresses as high as
possible. And do set it, DLLs with their base address still at 0x10000000
are embarrasing. I will be looking at those I package soon, there may be
one or two still left at low addresses.

–
#include <disclaimer.h></disclaimer.h>

6

Thanks for your feedback, very useful, i’ll do some further investigations
tomorrow

interesting you mentioned about how people forget to change the default dll
base address
of 0x10000000… i noticed yesterday that most of the dll’s in the latest
java runtime
envirnoment (1.4.2) are still set at 0x10000000! no wonder people are having
so many problems with
memory allocations in c code when using jni!! i think someone at sun
microsystems needs a
slap!

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Jakob Bohm
Sent: 10 June 2004 15:06
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Memory Mapped IO

On Thu, 10 Jun 2004 13:37:17 +0100, James Dunning
wrote:

> I have just noticed that when a process starts and loads all the
> necessary
> dll’s required for the process, the virtual address space in user-mode
> between 0x10000 - 07FFFFFFF is fragmented and in some case can even cause
> problems when you require to allocate a large chunk of contiguous memory.
> I don’t want to rebase all the system dll’s unless there is a windows
> system
> option or boot option which can force all dll’s to load at the top end of
> memory.
>
>
>
> I have a requirement where I need to be able to process extremely large
> data
> files which can be in excess of 20GB’s however I need to be able to map
> at
> least 512 Megabytes of the file into memory. Obviously I need to unmap
> and
> remap 512MB at various positions within the file.
>
>
> To do this I need to guarantee that I can remap 512MB once I have
> unmapped
> it, without other threads in the process fragmenting the previous mapped
> virtual address space!
>
>
> I thought I could achieve this by performing
>
>
> LPVOID pBaseAddress = VirtualAlloc(NULL, 0x20000000 MEM_RESERVE
> PAGE_READWRITE)
>
>
> Then passing the base address to MapViewOfFileEx but I cant seem to get
> it
> to work.
>

Its a small but simple mistake: Until you call VirtualFree to unreserve
that hunk of Virtual Address Space, Your own reservation will block
MapViewOfFileEx from using it (If it didn’t, DLLs could get loaded
there by mistake).

Possible solutions:

a) If you always have at least one 512MB hunk mapped, just delay your
call to UnMapViewOfFile to just before the MapViewOfFile call.

b) If there are periods where no 512MB hunk is mapped, do use VirtualAlloc
as suggested, but call VirtualFree (MEM_RELEASE) just before
MapViewOfFileEx
and VirtualAlloc(MEM_RESERVE) just after UnMapViewOfFile.

In either case, passing a specific base address to these calls is only
necessary if your code would fail at any other address. Otherwise leaving
that out increases your chance of success by allowing you to succeed if
something eats the head or tail of your allocation while still leaving
enough free space at the other end to allow it to go to a slightly
different
base address.

Also in either case make sure other threads in your own code isn’t trying
to
load dlls or allocate memory at the very moment of doing the free/reserve
pair of calls.

>
> Is there a way to guarantee that 512MB can be mapped once the previous
> section has been unmapped? Ideally I would like to reserve a chunk of
> memory
> for mapping my file then the process initially starts up.
>

In my experience the two biggest thieves of virtual addresses in the lowest
GB of process space are:
1. The stacks and heaps. If you do the reserve thing early enough
(Such as DllMain of a statically referenced Dll), there should be
just one stack and less than 3 heaps.

2. “Load in all processes” dlls that come with all sorts of
miscellaneous
utilities: Firewalls, Antivirus, Display Driver “extras”, East Asian
keyboard drivers from Microsoft, Advanced Mouse and Keyboard drivers
etc. Fortunately, many of those don’t load until at at least some of
the DllMain functions have been run.

Jakob

P.S.

A little plea to those here who put “load in all processes” dlls in their
drivers and utilities: Please try to set your base addresses as high as
possible. And do set it, DLLs with their base address still at 0x10000000
are embarrasing. I will be looking at those I package soon, there may be
one or two still left at low addresses.

–
#include <disclaimer.h>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@generaldynamics.uk.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

General Dynamics United Kingdom Limited
Registered in England and Wales No. 1911653
Registered Office: 100 New Bridge Street, London, EC4V 6JA</disclaimer.h>

7

I have another suggestion. If you repeatedly allocate and free your 512MB
of virtual memory and other memory allocation requests are occuring at the
same time then you are likely to fragment your usable memory range.
Assuming that runtime expands its heap from bottom up then your policy
may be to allocate your 512 MB from highest memory possible. This is an
option in VirtualAlloc call.

ned

James Dunning wrote:

Thanks for your feedback, very useful, i’ll do some further investigations
tomorrow

interesting you mentioned about how people forget to change the default dll
base address
of 0x10000000… i noticed yesterday that most of the dll’s in the latest
java runtime
envirnoment (1.4.2) are still set at 0x10000000! no wonder people are having
so many problems with
memory allocations in c code when using jni!! i think someone at sun
microsystems needs a
slap!

8

> of 0x10000000… i noticed yesterday that most of the dll’s in the latest

java runtime
envirnoment (1.4.2) are still set at 0x10000000! no wonder people are having

Ha-ha-ha! Good luck to Sun to beat the .NET’s performance :slight_smile:

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

9

> Ha-ha-ha! …

For the very same reason I buy Japanese car :). Now things are changing, so my next attempt would be home-made one :). But remember, Sun is now MS partner, Oracle is tending to move that way, but still might have some might against MS. An Apple is a long partner of MS, Isn’t it ?. SCO is under control, so Linux or SCO Unix would be the close to home-made !.

HP made a remarriage ( partly ) when got comapq. And IBM is fighting hard places some pawn(Linux) to block the rook attacks from MS, at least trying. An whole of ASIA is fighting for freedom ( get the Linux sourc code, tune it to their home grown, and be independent from Un-coutably infinite tons of MS’s iron clad hands). And not to distant past, MS and IBM were real buddies. May be some should really highlights what are commons in the kernel and windows design of OS/2 and NT. I’m least qulified for it, and have no sympathy of os/2. NT does not need sympathy.

Boy these days pop(s) might be very expensive, otherwise I need one or two, before streaching my mind to talk or think about these “sheer nonsense” And this is politics/economics at the corporate level, a guy with hardly 50 recurring bucks in his/her pocket should not give a flyin f* to all these. BTW, I only carry less than 50.
-pro