Mystery of "committing memory"??

NTDEV
1

Hi

I am trying to understand windows memory management mechanism. Everything was going very good until I tried to find out exact definition of committing memory.

-“Physical memory is a cache of the page file”
http://www.cs.washington.edu/education/courses/cse410/01au/lectures/cse410-27-VirtualMemory-6up.PDF

-“Memory backed by on-disk storage is called committed memory. Committed memory is backed either by page files, which can be dynamically resized, or by data/image files on secondary storage.”
Windows NT File System Internals, pg: 195

Here is what I understand so far:

1-) Reserving Phase: First we must reserve memory by calling VirtualAlloc (mem_reserve) which only builds VADs.

2-) Committing phase: When we call VirtualAlloc(Mem_Commit), Windows Memory Manager allocates space in the pagefile for the memory that we want to use. And builds PTEs and marks them invalid. But no physical memory allocation. So committing is just a disk operation.

3-) When program touches the memory that we committed in step 2, Page Fault occurs and Memory Manager brings the pages from disk to physical memory.

But Pavel Lebedinsky wrote in 2007:

"Committing an address range does not allocate any physical storage, it simply increases the number of committed pages in the system. "
http://www.tech-archive.net/Archive/Development/microsoft.public.win32.programmer.kernel/2007-10/msg00174.html

Now my question ;

Is committing memory, allocating space on disk or not? (But just on disk not memory, because some documents also say for committing memory, it is the process of allocating space either disk or physical memory. This is also one more confusing thing too. But this must be wrong, if there is an allocation it must be on disk not memory)

Please, can someone verify my steps and make exact definition of committing memory.

Thanks…

2

> Hi

I am trying to understand windows memory management mechanism. Everything
was going very good until I tried to find out exact definition of
committing memory.

-“Physical memory is a cache of the page file”
http://www.cs.washington.edu/education/courses/cse410/01au/lectures/cse410-27-VirtualMemory-6up.PDF

-“Memory backed by on-disk storage is called committed memory. Committed
memory is backed either by page files, which can be dynamically resized,
or by data/image files on secondary storage.”
Windows NT File System Internals, pg: 195

Here is what I understand so far:

1-) Reserving Phase: First we must reserve memory by calling VirtualAlloc
(mem_reserve) which only builds VADs.

2-) Committing phase: When we call VirtualAlloc(Mem_Commit), Windows
Memory Manager allocates space in the pagefile for the memory that we want
to use. And builds PTEs and marks them invalid. But no physical memory
allocation. So committing is just a disk operation.

Note that you can OR these flags together and do it all in one call.

3-) When program touches the memory that we committed in step 2, Page
Fault occurs and Memory Manager brings the pages from disk to physical
memory.

But Pavel Lebedinsky wrote in 2007:

"Committing an address range does not allocate any physical storage, it
simply increases the number of committed pages in the system. "
http://www.tech-archive.net/Archive/Development/microsoft.public.win32.programmer.kernel/2007-10/msg00174.html

Now my question ;

Is committing memory, allocating space on disk or not? (But just on disk
not memory, because some documents also say for committing memory, it is
the process of allocating space either disk or physical memory. This is
also one more confusing thing too. But this must be wrong, if there is an
allocation it must be on disk not memory)

It allocates space on the disk. If the space required exceeds the size of
your paging file, the allocation failss if it cannot get enough pages in
the pagefile.

Please, can someone verify my steps and make exact definition of
committing memory.

Thanks…

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

3

I think a given page doesn’t have a fixed address in the pagefile. The OS just makes sure the pagefile is big enough to find a page and to page it out.

When you commit memory, you make it accessible and usable. Uncommitted memory cannot be used by a program. It’s just an address range.

4

The three sources you refer to apply to Windows NT and are all over a decade
old. Having a pagefile in Windows (at least nowadays) is optional and so
must be all assumptions about disk backing of paged memory.

//Daniel

wrote in message news:xxxxx@ntdev…
> Hi
>
> I am trying to understand windows memory management mechanism. Everything
> was going very good until I tried to find out exact definition of
> committing memory.
>
> - “Windows NT requires “backing storage” for everything it keeps in RAM”
> http://support.microsoft.com/kb/99768
>
> -“Physical memory is a cache of the page file”
> http://www.cs.washington.edu/education/courses/cse410/01au/lectures/cse410-27-VirtualMemory-6up.PDF
>
> -“Memory backed by on-disk storage is called committed memory. Committed
> memory is backed either by page files, which can be dynamically resized,
> or by data/image files on secondary storage.”
> Windows NT File System Internals, pg: 195
>
> Here is what I understand so far:
>
>
> 1-) Reserving Phase: First we must reserve memory by calling VirtualAlloc
> (mem_reserve) which only builds VADs.
>
> 2-) Committing phase: When we call VirtualAlloc(Mem_Commit), Windows
> Memory Manager allocates space in the pagefile for the memory that we want
> to use. And builds PTEs and marks them invalid. But no physical memory
> allocation. So committing is just a disk operation.
>
> 3-) When program touches the memory that we committed in step 2, Page
> Fault occurs and Memory Manager brings the pages from disk to physical
> memory.
>
> But Pavel Lebedinsky wrote in 2007:
>
> "Committing an address range does not allocate any physical storage, it
> simply increases the number of committed pages in the system. "
> http://www.tech-archive.net/Archive/Development/microsoft.public.win32.programmer.kernel/2007-10/msg00174.html
>
> Now my question ;
>
> Is committing memory, allocating space on disk or not? (But just on disk
> not memory, because some documents also say for committing memory, it is
> the process of allocating space either disk or physical memory. This is
> also one more confusing thing too. But this must be wrong, if there is an
> allocation it must be on disk not memory)
>
> Please, can someone verify my steps and make exact definition of
> committing memory.
>
> Thanks…
>
>

5

i am afraid you got it wrong…

Taking into consideration that page allocated by VirtualAlloc() is zero-filled the first time it is accessed
(i.e. accessing it does not involve disk access), it does not really make sense to even think about disk space at the time of committing a page, because the target page is more than likely to be discarded long before the necessity to swap it to the disk arises. Committing a page simply means marking it committed, which means page fault handler has to bring the target page into RAM transparently to the failing code, no matter how long it takes (and it may take quite a while if the system is under memory pressure - the failing thread will have to block until memory becomes available). Otherwise, if page is not marked committed, page fault handler will throw an exception to the failing thread. Simple, ugh…

Anton Bassov

6

> 1-) Reserving Phase: First we must reserve memory by calling VirtualAlloc (mem_reserve) which only

builds VADs.

Reserve just reserves the address range, so it will not clash with some another allocation, stack, loaded DLL etc.

After reserve alone, you cannot touch these bytes.

Commit is what really allows you to touch the bytes. As about pagefile space allocation etc - this is all Mm’s innerworkings which you should not care (they change as Windows evolves).

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

7

First, I thank you all,

Daniel Terhell said that:
“Having a pagefile in Windows (at least nowadays) is optional and so must be all assumptions about disk backing of paged memory”

I am really curious about one thing. Yes, pagefile is optional, maybe someday it will be gone completely.

But after that how can memory sharing be implemented? I know all IPC methods use memory mapped files internally.

So that means memory sharing needs a disk file existence. Why? I think it is because we need a handle that represents the resource(I am not sure). And VirtualAlloc does not return a handle.

So without usage of a disk file, is it possible to implement memory sharing? If a new object, say “memory object” is defined, and VirtualAlloc returns a handle to that memory object, so can memory sharing be implemented without a disk file involvement??

Thanks again…

8

Pagefile backed sections are somewhat of a misnomer. The section is backed by commit charge on the machine - which could be either physical memory (RAM), or pagefile space.

The kernel knows how much storage space it has available to put memory allocations into (pagefile + RAM); this amount is the commitment quota available. Memory can be allocated (committed) for a pagefile backed section, or other user mode allocations, while there is free commit available.

  • S (Msft)

From: xxxxx@gmail.com
Sent: 11/19/2012 18:25
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Mystery of “committing memory”??

First, I thank you all,

Daniel Terhell said that:
“Having a pagefile in Windows (at least nowadays) is optional and so must be all assumptions about disk backing of paged memory”

I am really curious about one thing. Yes, pagefile is optional, maybe someday it will be gone completely.

But after that how can memory sharing be implemented? I know all IPC methods use memory mapped files internally.

So that means memory sharing needs a disk file existence. Why? I think it is because we need a handle that represents the resource(I am not sure). And VirtualAlloc does not return a handle.

So without usage of a disk file, is it possible to implement memory sharing? If a new object, say “memory object” is defined, and VirtualAlloc returns a handle to that memory object, so can memory sharing be implemented without a disk file involvement??

Thanks again…

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

9

Answers inline below…

First, I thank you all,

Daniel Terhell said that:
“Having a pagefile in Windows (at least nowadays) is optional and so must
be all assumptions about disk backing of paged memory”

I am really curious about one thing. Yes, pagefile is optional, maybe
someday it will be gone completely.

Actually, it is already gone today. Windows Embedded, or whatever the
package is currently called, requires enough RAM to hold all the processes
that need to run, because there is no disk at all.

Present or absent, its existence is irrelevant to the basic question; I
forget who posted the answer, but the notion that backing store is
required is a mere implementation decision is correct.

But after that how can memory sharing be implemented? I know all IPC
methods use memory mapped files internally.

Why do you think memory sharing requires memory-mapped files? In fact, it
does not. They are only one way to implement one kind of IPC, so how do
you jump to the conclusion that “all IPC methods use memory mapped files
internally”. What do you base this statement on?

So that means memory sharing needs a disk file existence. Why? I think it
is because we need a handle that represents the resource(I am not sure).
And VirtualAlloc does not return a handle.

No. Not at all. I’m not sure how it is done on embedded systems, but I
know that memory sharing does NOT require a disk file, because for the
last 15 years, one of the exercises in my Systems Programming course is
sharing memory and this exercise involves absolutely NO use of files.

So without usage of a disk file, is it possible to implement memory
sharing? If a new object, say “memory object” is defined, and VirtualAlloc
returns a handle to that memory object, so can memory sharing be
implemented without a disk file involvement??

Shared-segment DLLs, shared-segment executables. VirtualAlloc does not
participate in any of the memory sharing mechanisms I am aware of. And it
most definitely does not return a “handle” to a “memory object”. It
returns a process-local address to non-shared memory accessible to that
process (or NULL, if it fails). In fact, I have no idea why a disk file
would need to be involved at all. So you start with an unwarranted
assumption, and reach an unjustified conclusion. Even “memory-mapped
files” do not require an actual file; if the file given is
INVALID_HANDLE_VALUE, then the sharing is done without an actual disk
file. Older documents suggest that sharing is done via the pagefile, but
as already pointed out, this is an implementation detail, and is not
mandated by the abstract specification. So again, the notion of
memory-mapped “file” is not precluded by the absence of an actual “file”
system (or a writable file system, e.g., EPROM bulk memory).

So re-examine your assumptions. First, the assumption that “all IPC
requires shared memory” is clearly invalid (in the kernel, you can share
information by kernel addresses that is simply not accessible to user
space; it may be “shared” in an abstract sense, but there is absolutely no
actual “sharing”. WM_COPYDATA does not require a disk file system, and
can be used to transmit data from one process to another; whether this is
done by shared memory or some other mechanism is completely irrelevant to
its usage, and it is definitely an IPC mechanism. Registered Window
Messages can pass a WPARAM-sized and an LPARAM-sized data object, and
shared memory is definitely not required for this. You can use unnamed
pipes, named pipes, and mailslots as IPC mechanisms, and no shared memory
is involved. You could even use TCP/IP or UDP for IPC, and no shared
memory is required. I’m sure I can come up with a few more ideas for IPC
that do not require shared memory, so I fail to see how you have leapt to
the conclusion that shared memory is necessary to all IPC mechanisms.
Then the assumption that “a pagefile is required for shared memory” is
seriously questionable. So if you throw these two assumptions out, there
is no problem.
joe

Thanks again…

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

10

> But after that how can memory sharing be implemented?

How is this connected to pagefiles?

I know all IPC methods use memory mapped files internally.

Not so. LPC messages not such. Named pipes are not such.

So without usage of a disk file, is it possible to implement memory sharing?

Surely yes.

If a new object, say “memory object” is defined

It is a section object, created by CreateFileMapping.

If you call this function with NULL file handle, then you have a “memory object” not associated with the disk file.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

11

>Pagefile backed sections are somewhat of a misnomer.

The whole phrase of “pagefile-backed section” is obsolete and only reflects the implementation details of some older Windows.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

12

Joseph M. Newcomer :
“Why do you think memory sharing requires memory-mapped files?”

1-) “You can use memory-mapped files to allow multiple processes running on the same machine to share data with each other. Windows does offer other methods for communicating data among processes?but these other methods are implemented using memory-mapped files, making memory-mapped files the most efficient way for multiple processes on a single machine to communicate with one another.”
Windows via C/C++

2-) “Multiple processes can share memory by mapping their virtual address spaces to the same file or to the paging file (interprocess memory sharing is the principal reason for mapping to the paging file).”
Windows System Programming

Joseph M. Newcomer :
“Even “memory-mapped files” do not require an actual file; if the file given is INVALID_HANDLE_VALUE, then the sharing is done without an actual disk file. Older documents suggest that sharing is done via the pagefile, but as already pointed out”

So that means Microsoft must update its whole documentation of memory-mapped files.

“If hFile is INVALID_HANDLE_VALUE, the calling process must also specify a size for the file mapping object in the dwMaximumSizeHigh and dwMaximumSizeLow parameters. In this scenario, CreateFileMapping creates a file mapping object of a specified size that is backed by the system paging file instead of by a file in the file system.”
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537(v=vs.85).aspx

13

>but these other methods are implemented using memory-mapped files

Nonsense. Named pipes are not such. LPC is not such.

, making memory-mapped files the most efficient way

Efficient? what about the locking issues with them?

Bad and inefficient (in terms of development and bugfixing time) way.

RPC/DCOM/.NET Remoting are the ways to go.

Windows via C/C++

Bad book. Will never buy one.

2-) "Multiple processes can share memory by mapping their virtual address spaces to the same file
or to the paging file (interprocess memory sharing is the principal reason for mapping to the paging

“Mapping to paging file” is an obsolete notion which reflects some implementation details in Windows NT 3.x or such.

The real name for this technique is “anonymous mapping”, i.e. section object not tied to a file.

So that means Microsoft must update its whole documentation of memory-mapped files.

Yes it should. The wording of “pagefile-backed section object” is evil and misleading.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

14

xxxxx@flounder.com wrote:

Actually, it is already gone today. Windows Embedded, or whatever the
package is currently called, requires enough RAM to hold all the processes
that need to run, because there is no disk at all.

Well, there is a naming confusion here. “Windows Compact Embedded”, the
product formerly known as Windows CE, can run entirely from RAM.
“Windows Embedded”, the micro-configurable version of the desktop
systems, is virtually indistinguishable from the desktop systems, and
runs with a disk.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

15

> Joseph M. Newcomer :

“Why do you think memory sharing requires memory-mapped files?”

1-) “You can use memory-mapped files to allow multiple processes running
on the same machine to share data with each other. Windows does offer
other methods for communicating data among processes?but these other
methods are implemented using memory-mapped files, making memory-mapped
files the most efficient way for multiple processes on a single machine to
communicate with one another.”
Windows via C/C++

The above statement is erroneous. The other methods are most definitely
NOT implemented using memory-mapped files, and the author has made a
clearly false statement.

And, surprisingly, it is not the “most efficient” way to do IPC; it is
merely the most efficient way to share LARGE address spaces. If I have a
small number of bits, I would look at named pipes, anonymous pipes,
WM_COPYDATA and Registered Window Messages as better means for sharing
small blocks of data.

2-) “Multiple processes can share memory by mapping their virtual address
spaces to the same file or to the paging file (interprocess memory sharing
is the principal reason for mapping to the paging file).”
Windows System Programming

Yes, the principal reason for mapping a file into more than one process is
to implement interprocess memory sharing, and general memory sharing is
often accomplished by using a named mapping not associated with a file
(and, as already pointed out, was implemented in earlier versions of
Windows by using the paging file), but statement 1 is so clearly wrong
that it simply does not follow that all possible IPC mechanisms must use
file sharing. Using shared memory mapping is one of MANY ways to
implement IPC, and as far as I know, is the ONLY mechanism that is
implemented using shared memory.

Joseph M. Newcomer :
“Even “memory-mapped files” do not require an actual file; if the file
given is INVALID_HANDLE_VALUE, then the sharing is done without an actual
disk file. Older documents suggest that sharing is done via the pagefile,
but as already pointed out”

So that means Microsoft must update its whole documentation of
memory-mapped files.

Yep.

“If hFile is INVALID_HANDLE_VALUE, the calling process must also specify a
size for the file mapping object in the dwMaximumSizeHigh and
dwMaximumSizeLow parameters. In this scenario, CreateFileMapping creates a
file mapping object of a specified size that is backed by the system
paging file instead of by a file in the file system.”
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537(v=vs.85).aspx

As others have pointed out, this is no longer true, and the documentation
is therefore obsolete.
joe

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

16

OK. Thanks, I understand it.

Also beware of Windows via C/C++ book’s section where Memory Mapped File is explained because author claims, multiple times through the book, that all of the IPC methods use memory-mapped file. Just a note for members of this group. Thanks…

“Windows has always excelled at offering mechanisms that allow applications to share data and information quickly and easily. These mechanisms include RPC, COM, OLE, DDE, window messages (especially WM_COPYDATA), the Clipboard, mailslots, pipes, sockets, and so on. In Windows, the lowest-level mechanism for sharing data on a single machine is the memory-mapped file. That’s right, all of the mechanisms I mention ultimately use memory-mapped files to do their dirty work if all the processes communicating are on the same machine”
http://my.safaribooksonline.com/book/programming/visual-cplusplus/9780735639904/memory-mapped-files/using_memory-mapped_files_to_share_data

17

It surprises you that some random dude is allowed to author a technical book, and that such a book is filled with lots of incorrect information? REALLY?

I hate to burst your bubble, but take it from me: There’s really no serious vetting of the technical qualifications of those who author technical books. The publishers don’t care. What they want are authors who (a) write manuscript pages according to their rules, (b) produce such pages at a steady, regular, and pre-agreed pace, (c) will accept little money for their work, (d) constantly act grateful that a publisher would even CONSIDER publishing taking output and put it in a book, and (e) make as close to zero demands about ANYthing whatsoever regarding the book – that includes demands about illustrations, indexing, layout, production, editing, or promotion.

When it comes to technical books, they don’t want quality content. They want content that will sell on Amazon.

If they thought they could sell 24 chapters with 800 pages of Lorem Ipsum with a title that reads “.Net UNMASKED!” most publishers would be more than happy to publish it.

I’m sure there are exceptions, of course. And the above does not apply to text books, which are a whole different game.

Peter
OSR

18

> small number of bits, I would look at named pipes, anonymous pipes,

No anon pipes in Windows, they are actually named pipes with autogenerated names.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

19

But that is just an implementation detail. CreatePipe creates two handles
to an anonymous pipe, so from the viewpoint of a programmer, and the Win32
API, anonymous pipes are quite real, and different from named pipes in
their behavior. Therefore, they are a valid concept to use for IPC.
joe

> small number of bits, I would look at named pipes, anonymous pipes,

No anon pipes in Windows, they are actually named pipes with autogenerated
names.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer