memory management / paging question

NTFSD
1

Hi folks. I was wondering if anyone could help me with some memory
management/paging questions. I’m afraid its a bit more involved than a one
liner.

I’m using virtual memory in the address space of my companion service. The
virtual address is backed by a memory mapped files, as opposed to the
pagefile, if it matters.

Early on I do IoAllocateMdl then MmInitializeMdl for this UserAddress. (In
fact I have a bunch of them, and they are used for passing information to
the companion service.)

pMdl = IoAllocateMdl ( UserAddress, Size, FALSE, FALSE, NULL );
MmInitializeMdl( pMdl, UserAddress, Size );

When I want to use this virtual memory I do MmProbeAndLockPages then
MmGetSystemAddressForMdlSafe.

MmProbeAndLockPages( pMdl, KernelMode, IoWriteAccess );
SystemAddress = MmGetSystemAddressForMdlSafe( pMdl, HighPagePriority );

If I understand this right, then if this virtual memory is not in-memory,
is in the backing file, then the MmProbeAndLockPages will cause page
reads. I really want to avoid these reads since they dont contain useful
data. So the first question is how can I avoid these page reads at this
point?

When the driver has filled this memory with data for the companion
service, it unlocks the pages with MmUnlockPages, and ‘passes’ it to the
companion using a reverse iotcl.

MmUnlockPages( pMdl );

Sometime after this the companion service has done with the data and
‘passes’ it to the driver using an ioctl. If all of this happened quickly
enough the memory manager might not have written the dirty pages to the
backing file. But, it might do that from this point forward until the
driver next uses the memory. I want to avoid that, because it doesnt serve
any purpose for me. So the second question is how can I avoid these page
writes at this point?

For completeness, in the end the Mdl is just freed with IoFreeMdl.
IoFreeMdl ( pMdl );

Thanks in advance
Lyndon

2

Lyndon,

If the ValidDataLength of the file is below the region you are reading, all
that should happen is you get zero filled paged (not from disk) which will
avoid disk I/O to page in the initial pages.

If you don’t want the pages to be written back out, leave them locked until
you are done with them. So long as they are locked, they will remain in
memory. Unless we are talking about a significant amount of memory, locking
it down should not be an issue.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Lyndon J. Clarke [mailto:xxxxx@gcplc.com]
Sent: Wednesday, March 05, 2003 10:04 AM
To: File Systems Developers
Subject: [ntfsd] memory management / paging question

Hi folks. I was wondering if anyone could help me with some memory
management/paging questions. I’m afraid its a bit more involved than a one
liner.

I’m using virtual memory in the address space of my companion service. The
virtual address is backed by a memory mapped files, as opposed to the
pagefile, if it matters.

Early on I do IoAllocateMdl then MmInitializeMdl for this UserAddress. (In
fact I have a bunch of them, and they are used for passing information to
the companion service.)

pMdl = IoAllocateMdl ( UserAddress, Size, FALSE, FALSE, NULL );
MmInitializeMdl( pMdl, UserAddress, Size );

When I want to use this virtual memory I do MmProbeAndLockPages then
MmGetSystemAddressForMdlSafe.

MmProbeAndLockPages( pMdl, KernelMode, IoWriteAccess );
SystemAddress = MmGetSystemAddressForMdlSafe( pMdl, HighPagePriority );

If I understand this right, then if this virtual memory is not in-memory,
is in the backing file, then the MmProbeAndLockPages will cause page
reads. I really want to avoid these reads since they dont contain useful
data. So the first question is how can I avoid these page reads at this
point?

When the driver has filled this memory with data for the companion
service, it unlocks the pages with MmUnlockPages, and ‘passes’ it to the
companion using a reverse iotcl.

MmUnlockPages( pMdl );

Sometime after this the companion service has done with the data and
‘passes’ it to the driver using an ioctl. If all of this happened quickly
enough the memory manager might not have written the dirty pages to the
backing file. But, it might do that from this point forward until the
driver next uses the memory. I want to avoid that, because it doesnt serve
any purpose for me. So the second question is how can I avoid these page
writes at this point?

For completeness, in the end the Mdl is just freed with IoFreeMdl.
IoFreeMdl ( pMdl );

Thanks in advance
Lyndon

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

3

Hi Tony

Thanks for your quick response. The ValidDataLength is not below the
region I am using here. I cannot keep the pages locked down while the
companion service is doing its thing, since that might take a long time,
and we could be talking about large amounts of memory here. Hence my
rather focussed but long in the details post.

Kind regards
Lyndon

4

> If I understand this right, then if this virtual memory is not
in-memory,

is in the backing file, then the MmProbeAndLockPages will cause page
reads. I really want to avoid these reads since they dont contain
useful
data. So the first question is how can I avoid these page reads at
this
point?

Mm will avoid disk reads for virgin (just allocated and never written
to) pages. It will pick them from zero page list.

Max

5

Hi,

when you ask for IoWriteAccess, to my understanding you’re declaring that you’re only going to write to those pages and that you don’t care about the original contents. The sensible thing for the MM to do in this case would be to back any non-resident whole pages in the MDL from uninitialized pages - to back it with zero-filled pages is a waste.
I don’t know if this is in fact the case.

Siggi

-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Wednesday, March 05, 2003 10:15 AM
To: File Systems Developers
Subject: [ntfsd] RE: memory management / paging question

Lyndon,

If the ValidDataLength of the file is below the region you are reading, all
that should happen is you get zero filled paged (not from disk) which will
avoid disk I/O to page in the initial pages.

If you don’t want the pages to be written back out, leave them locked until
you are done with them. So long as they are locked, they will remain in
memory. Unless we are talking about a significant amount of memory, locking
it down should not be an issue.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Lyndon J. Clarke [mailto:xxxxx@gcplc.com]
Sent: Wednesday, March 05, 2003 10:04 AM
To: File Systems Developers
Subject: [ntfsd] memory management / paging question

Hi folks. I was wondering if anyone could help me with some memory
management/paging questions. I’m afraid its a bit more involved than a one
liner.

I’m using virtual memory in the address space of my companion service. The
virtual address is backed by a memory mapped files, as opposed to the
pagefile, if it matters.

Early on I do IoAllocateMdl then MmInitializeMdl for this UserAddress. (In
fact I have a bunch of them, and they are used for passing information to
the companion service.)

pMdl = IoAllocateMdl ( UserAddress, Size, FALSE, FALSE, NULL );
MmInitializeMdl( pMdl, UserAddress, Size );

When I want to use this virtual memory I do MmProbeAndLockPages then
MmGetSystemAddressForMdlSafe.

MmProbeAndLockPages( pMdl, KernelMode, IoWriteAccess );
SystemAddress = MmGetSystemAddressForMdlSafe( pMdl, HighPagePriority );

If I understand this right, then if this virtual memory is not in-memory,
is in the backing file, then the MmProbeAndLockPages will cause page
reads. I really want to avoid these reads since they dont contain useful
data. So the first question is how can I avoid these page reads at this
point?

When the driver has filled this memory with data for the companion
service, it unlocks the pages with MmUnlockPages, and ‘passes’ it to the
companion using a reverse iotcl.

MmUnlockPages( pMdl );

Sometime after this the companion service has done with the data and
‘passes’ it to the driver using an ioctl. If all of this happened quickly
enough the memory manager might not have written the dirty pages to the
backing file. But, it might do that from this point forward until the
driver next uses the memory. I want to avoid that, because it doesnt serve
any purpose for me. So the second question is how can I avoid these page
writes at this point?

For completeness, in the end the Mdl is just freed with IoFreeMdl.
IoFreeMdl ( pMdl );

Thanks in advance
Lyndon

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntfsd as: xxxxx@greenborder.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

6

My first thought as to why is for security reasons, the pages are zero
filled. What if there happened to be some ‘important’ data in the pages
that you mapped to and all you did was look at the data in these pages
w/o them being zero filled? This, in my understanding, would be a big
hole in the security model of MS.

Pete

Peter Scott
xxxxx@KernelDrivers.com
www.KernelDrivers.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Sigurdur
Asgeirsson
Sent: Sunday, March 09, 2003 2:45 PM
To: File Systems Developers
Subject: [ntfsd] RE: memory management / paging question

Hi,

when you ask for IoWriteAccess, to my understanding you’re declaring
that you’re only going to write to those pages and that you don’t care
about the original contents. The sensible thing for the MM to do in this
case would be to back any non-resident whole pages in the MDL from
uninitialized pages - to back it with zero-filled pages is a waste.
I don’t know if this is in fact the case.

Siggi

-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Wednesday, March 05, 2003 10:15 AM
To: File Systems Developers
Subject: [ntfsd] RE: memory management / paging question

Lyndon,

If the ValidDataLength of the file is below the region you are reading,
all
that should happen is you get zero filled paged (not from disk) which
will
avoid disk I/O to page in the initial pages.

If you don’t want the pages to be written back out, leave them locked
until
you are done with them. So long as they are locked, they will remain in
memory. Unless we are talking about a significant amount of memory,
locking
it down should not be an issue.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Lyndon J. Clarke [mailto:xxxxx@gcplc.com]
Sent: Wednesday, March 05, 2003 10:04 AM
To: File Systems Developers
Subject: [ntfsd] memory management / paging question

Hi folks. I was wondering if anyone could help me with some memory
management/paging questions. I’m afraid its a bit more involved than a
one
liner.

I’m using virtual memory in the address space of my companion service.
The
virtual address is backed by a memory mapped files, as opposed to the
pagefile, if it matters.

Early on I do IoAllocateMdl then MmInitializeMdl for this UserAddress.
(In
fact I have a bunch of them, and they are used for passing information
to
the companion service.)

pMdl = IoAllocateMdl ( UserAddress, Size, FALSE, FALSE, NULL );
MmInitializeMdl( pMdl, UserAddress, Size );

When I want to use this virtual memory I do MmProbeAndLockPages then
MmGetSystemAddressForMdlSafe.

MmProbeAndLockPages( pMdl, KernelMode, IoWriteAccess );
SystemAddress = MmGetSystemAddressForMdlSafe( pMdl, HighPagePriority );

If I understand this right, then if this virtual memory is not
in-memory,
is in the backing file, then the MmProbeAndLockPages will cause page
reads. I really want to avoid these reads since they dont contain useful
data. So the first question is how can I avoid these page reads at this
point?

When the driver has filled this memory with data for the companion
service, it unlocks the pages with MmUnlockPages, and ‘passes’ it to the
companion using a reverse iotcl.

MmUnlockPages( pMdl );

Sometime after this the companion service has done with the data and
‘passes’ it to the driver using an ioctl. If all of this happened
quickly
enough the memory manager might not have written the dirty pages to the
backing file. But, it might do that from this point forward until the
driver next uses the memory. I want to avoid that, because it doesnt
serve
any purpose for me. So the second question is how can I avoid these
page
writes at this point?

For completeness, in the end the Mdl is just freed with IoFreeMdl.
IoFreeMdl ( pMdl );

Thanks in advance
Lyndon

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntfsd as: xxxxx@greenborder.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntfsd as: xxxxx@kerneldrivers.com
To unsubscribe send a blank email to xxxxx@lists.osr.com