Memory Mapped Interface

NTFSD
1

Is there a way to tell the kernel that your FSD does not support memory
mapped files? Even though this may not be a correct design, I am just
curious.

We have found that notepad memory maps a file in Windows 2000. So, our
issue of not supporting memory mapped files has surfaced as we are porting
from Windows NT to Windows 2000.

I did find a very interesting qoute on the list server( Thanks Tony ), “A
GREAT trick is to open a file, memory map it, close the file, and use the
memory mapping to modify the file contents. THAT is the case which causes
the greatest problem to most file systems and file system filter drivers.”

One of our assumptions was that a IRP_MJ_CLEANUP would be followed by a
IRP_MJ_CLOSE, which appeared to happen on NT. However, I now realize that
in Windows 2000 the IRP_MJ_CLOSE can be deferred to a later point in time
by the Memory Manager. In the case of a memory mapped file we are seeing
:

IRP_MJ_CREATE
IRP_MJ_READ
IRP_MJ_CLEANUP
IRP_MJ_READ
IRP_MJ_CLOSE

Our current design fails on the second IRP_MJ_READ, since we assumed a
IRP_MJ_CLOSE would follow the IRP_MJ_CLEANUP.

2

Yes, don’t fill in FileObject->SectionObjectPointers.

This is actually how usermode is prevented from staring at the contents
of directories. We create a second stream file object, which doesn’t
have a usermode handle, fill in *its* SectionObjectPointers and use it
for caching the data.

I expect this’ll provoke an AHA! moment when looking at the IFSKIT
source.

The question of how this will affect apps which expect memory mapping to
work, of which notepad is only a cursory example, I’ll have to leave to
your expertise/scenario if it is not the general case. If it is, you
should support mapping. Note though that prohibiting memory mapping does
not prevent other operations from coming in between cleanup and close,
the most straightforward example of which is the lazy writer pushing
dirty data (produced by good old WriteFile) out after the cleanup.

-----Original Message-----
From: xxxxx@riolabs.com [mailto:xxxxx@riolabs.com]
Sent: Sunday, November 26, 2000 5:00 PM
To: File Systems Developers
Subject: [ntfsd] Memory Mapped Interface

Is there a way to tell the kernel that your FSD does not support memory
mapped files? Even though this may not be a correct design, I am just
curious.

We have found that notepad memory maps a file in Windows 2000. So, our
issue of not supporting memory mapped files has surfaced as we are
porting
from Windows NT to Windows 2000.

I did find a very interesting qoute on the list server( Thanks Tony ),
“A
GREAT trick is to open a file, memory map it, close the file, and use
the
memory mapping to modify the file contents. THAT is the case which
causes
the greatest problem to most file systems and file system filter
drivers.”

One of our assumptions was that a IRP_MJ_CLEANUP would be followed by a
IRP_MJ_CLOSE, which appeared to happen on NT. However, I now realize
that
in Windows 2000 the IRP_MJ_CLOSE can be deferred to a later point in
time
by the Memory Manager. In the case of a memory mapped file we are
seeing
:

IRP_MJ_CREATE
IRP_MJ_READ
IRP_MJ_CLEANUP
IRP_MJ_READ
IRP_MJ_CLOSE

Our current design fails on the second IRP_MJ_READ, since we assumed a
IRP_MJ_CLOSE would follow the IRP_MJ_CLEANUP.

You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
To unsubscribe send a blank email to $subst(‘Email.Unsub’)

3

> One of our assumptions was that a IRP_MJ_CLEANUP would be followed by a

IRP_MJ_CLOSE, which appeared to happen on NT. However, I now realize that
in Windows 2000 the IRP_MJ_CLOSE can be deferred to a later point in

In the case described by Tony, the sequence will be
CLEANUP
the file pages changed
several WRITE from the MPW thread or from MmFlushSection
CLOSE

Max

4

Actually, if you disable memory mapping, you would (effectively) disable the
lazy writer as well, since without memory mapping, there can be no caching
(via the cache manager) and hence the lazy writer would never be invoked for
the given file.

Regards,

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

“Daniel Lovinger” wrote in message
news:xxxxx@ntfsd…
>
> Yes, don’t fill in FileObject->SectionObjectPointers.
>
> This is actually how usermode is prevented from staring at the contents
> of directories. We create a second stream file object, which doesn’t
> have a usermode handle, fill in its SectionObjectPointers and use it
> for caching the data.
>
> I expect this’ll provoke an AHA! moment when looking at the IFSKIT
> source.
>
> The question of how this will affect apps which expect memory mapping to
> work, of which notepad is only a cursory example, I’ll have to leave to
> your expertise/scenario if it is not the general case. If it is, you
> should support mapping. Note though that prohibiting memory mapping does
> not prevent other operations from coming in between cleanup and close,
> the most straightforward example of which is the lazy writer pushing
> dirty data (produced by good old WriteFile) out after the cleanup.
>
> -----Original Message-----
> From: xxxxx@riolabs.com [mailto:xxxxx@riolabs.com]
> Sent: Sunday, November 26, 2000 5:00 PM
> To: File Systems Developers
> Subject: [ntfsd] Memory Mapped Interface
>
>
> Is there a way to tell the kernel that your FSD does not support memory
> mapped files? Even though this may not be a correct design, I am just
> curious.
>
> We have found that notepad memory maps a file in Windows 2000. So, our
> issue of not supporting memory mapped files has surfaced as we are
> porting
> from Windows NT to Windows 2000.
>
> I did find a very interesting qoute on the list server( Thanks Tony ),
> “A
> GREAT trick is to open a file, memory map it, close the file, and use
> the
> memory mapping to modify the file contents. THAT is the case which
> causes
> the greatest problem to most file systems and file system filter
> drivers.”
>
> One of our assumptions was that a IRP_MJ_CLEANUP would be followed by a
> IRP_MJ_CLOSE, which appeared to happen on NT. However, I now realize
> that
> in Windows 2000 the IRP_MJ_CLOSE can be deferred to a later point in
> time
> by the Memory Manager. In the case of a memory mapped file we are
> seeing
> :
>
> IRP_MJ_CREATE
> IRP_MJ_READ
> IRP_MJ_CLEANUP
> IRP_MJ_READ
> IRP_MJ_CLOSE
>
> Our current design fails on the second IRP_MJ_READ, since we assumed a
> IRP_MJ_CLOSE would follow the IRP_MJ_CLEANUP.
>
> —
> You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
> To unsubscribe send a blank email to $subst(‘Email.Unsub’)
>
>
>
>

5

Between IRP_MJ_CLEANUP and IRP_MJ_CLOSE the only operations you should
legitimately observe are those related to paging. Thus, you can see:

IRP_MJ_READ - processing a page fault
IRP_MJ_WRITE - storing dirty pages back to the backing store (disk, network,
paper tape, whatever…)
IRP_MJ_GET_INFORMATION - retrieving file size information
IRP_MJ_SET_INFORMATION - ensuring the EOF is set properly

That’s my recollection at least.

Regards,

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

wrote in message news:xxxxx@ntfsd…
>
> Is there a way to tell the kernel that your FSD does not support memory
> mapped files? Even though this may not be a correct design, I am just
> curious.
>
> We have found that notepad memory maps a file in Windows 2000. So, our
> issue of not supporting memory mapped files has surfaced as we are porting
> from Windows NT to Windows 2000.
>
> I did find a very interesting qoute on the list server( Thanks Tony ), “A
> GREAT trick is to open a file, memory map it, close the file, and use the
> memory mapping to modify the file contents. THAT is the case which causes
> the greatest problem to most file systems and file system filter drivers.”
>
> One of our assumptions was that a IRP_MJ_CLEANUP would be followed by a
> IRP_MJ_CLOSE, which appeared to happen on NT. However, I now realize that
> in Windows 2000 the IRP_MJ_CLOSE can be deferred to a later point in time
> by the Memory Manager. In the case of a memory mapped file we are seeing
> :
>
> IRP_MJ_CREATE
> IRP_MJ_READ
> IRP_MJ_CLEANUP
> IRP_MJ_READ
> IRP_MJ_CLOSE

>
> Our current design fails on the second IRP_MJ_READ, since we assumed a
> IRP_MJ_CLOSE would follow the IRP_MJ_CLEANUP.
>
>

6

Yet, the converse of this appears to work (right Tony?). To solve the exact
Windows 2000 Notepad problem described below, we implemented the VMM
interface without the Cache Mgr. For now, anyway.

Russ

-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Monday, November 27, 2000 11:16 PM
To: File Systems Developers
Subject: [ntfsd] Re: Memory Mapped Interface

Actually, if you disable memory mapping, you would (effectively) disable the
lazy writer as well, since without memory mapping, there can be no caching
(via the cache manager) and hence the lazy writer would never be invoked for
the given file.

Regards,

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

“Daniel Lovinger” wrote in message
news:xxxxx@ntfsd…
>
> Yes, don’t fill in FileObject->SectionObjectPointers.
>
> This is actually how usermode is prevented from staring at the contents
> of directories. We create a second stream file object, which doesn’t
> have a usermode handle, fill in its SectionObjectPointers and use it
> for caching the data.
>
> I expect this’ll provoke an AHA! moment when looking at the IFSKIT
> source.
>
> The question of how this will affect apps which expect memory mapping to
> work, of which notepad is only a cursory example, I’ll have to leave to
> your expertise/scenario if it is not the general case. If it is, you
> should support mapping. Note though that prohibiting memory mapping does
> not prevent other operations from coming in between cleanup and close,
> the most straightforward example of which is the lazy writer pushing
> dirty data (produced by good old WriteFile) out after the cleanup.
>
> -----Original Message-----
> From: xxxxx@riolabs.com [mailto:xxxxx@riolabs.com]
> Sent: Sunday, November 26, 2000 5:00 PM
> To: File Systems Developers
> Subject: [ntfsd] Memory Mapped Interface
>
>
> Is there a way to tell the kernel that your FSD does not support memory
> mapped files? Even though this may not be a correct design, I am just
> curious.
>
> We have found that notepad memory maps a file in Windows 2000. So, our
> issue of not supporting memory mapped files has surfaced as we are
> porting
> from Windows NT to Windows 2000.
>
> I did find a very interesting qoute on the list server( Thanks Tony ),
> “A
> GREAT trick is to open a file, memory map it, close the file, and use
> the
> memory mapping to modify the file contents. THAT is the case which
> causes
> the greatest problem to most file systems and file system filter
> drivers.”
>
> One of our assumptions was that a IRP_MJ_CLEANUP would be followed by a
> IRP_MJ_CLOSE, which appeared to happen on NT. However, I now realize
> that
> in Windows 2000 the IRP_MJ_CLOSE can be deferred to a later point in
> time
> by the Memory Manager. In the case of a memory mapped file we are
> seeing
> :
>
> IRP_MJ_CREATE
> IRP_MJ_READ
> IRP_MJ_CLEANUP
> IRP_MJ_READ
> IRP_MJ_CLOSE
>
> Our current design fails on the second IRP_MJ_READ, since we assumed a
> IRP_MJ_CLOSE would follow the IRP_MJ_CLEANUP.
>
> —
> You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
> To unsubscribe send a blank email to $subst(‘Email.Unsub’)
>
>
>
>

—
You are currently subscribed to ntfsd as: xxxxx@ENGENIA.COM
To unsubscribe send a blank email to $subst(‘Email.Unsub’)

7

Hi Russ,

I would like to get a hint from you how to have a working file mapping
without using NT Cache manager ?
I have a FSD working fine as for read/write open/close and directory list
under Windows Explorer , but I cannot execute
a .EXE file . When I set FileObject->SectionObjectPointer =
&(Fcb->NTRequiredFCB.SectionObject) the MM start working
because I get 2 IRP_MJ_READs with IRP_PAGING_IO set at the Irp->Flags ,
but nothing more happens when I double click on a .EXE file
placed in my file system . No error message and no hangup happens .
If I avoid setting the FileObject->SectionObjectPointer and I double
click a .EXE file I get a message : " Not a windows NT file …"

Can you elaborate on how you succeed in running a .EXE file ( actually a
mapped file) by your NT FSD without using the NT cache manager ?

thanks,

Dubi

8

The disadvantage of implementing the VMM interface without the cache manager
is that you have a consistency “problem” that you must solve (or ignore.
That’s ONE “solution” - just ignore consistency.)

But yes, it does work. It just isn’t a solution I personally like.
Something about working on coherent distributed file sysems for so many
years…

Regards,

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Russ Auerbach [mailto:xxxxx@ENGENIA.COM]
Sent: Tuesday, November 28, 2000 8:48 AM
To: File Systems Developers
Subject: [ntfsd] Re: Memory Mapped Interface

Yet, the converse of this appears to work (right Tony?). To solve the exact
Windows 2000 Notepad problem described below, we implemented the VMM
interface without the Cache Mgr. For now, anyway.

Russ

-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Monday, November 27, 2000 11:16 PM
To: File Systems Developers
Subject: [ntfsd] Re: Memory Mapped Interface

Actually, if you disable memory mapping, you would (effectively) disable the
lazy writer as well, since without memory mapping, there can be no caching
(via the cache manager) and hence the lazy writer would never be invoked for
the given file.

Regards,

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

“Daniel Lovinger” wrote in message
news:xxxxx@ntfsd…
>
> Yes, don’t fill in FileObject->SectionObjectPointers.
>
> This is actually how usermode is prevented from staring at the contents
> of directories. We create a second stream file object, which doesn’t
> have a usermode handle, fill in its SectionObjectPointers and use it
> for caching the data.
>
> I expect this’ll provoke an AHA! moment when looking at the IFSKIT
> source.
>
> The question of how this will affect apps which expect memory mapping to
> work, of which notepad is only a cursory example, I’ll have to leave to
> your expertise/scenario if it is not the general case. If it is, you
> should support mapping. Note though that prohibiting memory mapping does
> not prevent other operations from coming in between cleanup and close,
> the most straightforward example of which is the lazy writer pushing
> dirty data (produced by good old WriteFile) out after the cleanup.
>
> -----Original Message-----
> From: xxxxx@riolabs.com [mailto:xxxxx@riolabs.com]
> Sent: Sunday, November 26, 2000 5:00 PM
> To: File Systems Developers
> Subject: [ntfsd] Memory Mapped Interface
>
>
> Is there a way to tell the kernel that your FSD does not support memory
> mapped files? Even though this may not be a correct design, I am just
> curious.
>
> We have found that notepad memory maps a file in Windows 2000. So, our
> issue of not supporting memory mapped files has surfaced as we are
> porting
> from Windows NT to Windows 2000.
>
> I did find a very interesting qoute on the list server( Thanks Tony ),
> “A
> GREAT trick is to open a file, memory map it, close the file, and use
> the
> memory mapping to modify the file contents. THAT is the case which
> causes
> the greatest problem to most file systems and file system filter
> drivers.”
>
> One of our assumptions was that a IRP_MJ_CLEANUP would be followed by a
> IRP_MJ_CLOSE, which appeared to happen on NT. However, I now realize
> that
> in Windows 2000 the IRP_MJ_CLOSE can be deferred to a later point in
> time
> by the Memory Manager. In the case of a memory mapped file we are
> seeing
> :
>
> IRP_MJ_CREATE
> IRP_MJ_READ
> IRP_MJ_CLEANUP
> IRP_MJ_READ
> IRP_MJ_CLOSE
>
> Our current design fails on the second IRP_MJ_READ, since we assumed a
> IRP_MJ_CLOSE would follow the IRP_MJ_CLEANUP.
>
> —
> You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
> To unsubscribe send a blank email to $subst(‘Email.Unsub’)
>
>
>
>

—
You are currently subscribed to ntfsd as: xxxxx@ENGENIA.COM
To unsubscribe send a blank email to $subst(‘Email.Unsub’)

—
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to $subst(‘Email.Unsub’)