A good source of information on MEMORY MAPPED FILES!!

NTFSD
1

Hi!
Does anyone know where can i find information on Memory mapped files??
I am not very clear with this concept…

2

Sorry for being a %RudeWord%, but I really can’t help myself.
A few “memory mapped files” guys asked the same questions
in last two weeks.

If you “cannot find” any info on it, you probably don’t know
how to use google. Let me try a few “advanced tricks”.

I opened Google in my browser, and entered “memory mapped files”.
Pressed Enter. The first found link explains something about
memory mapped files, but seems to be Linux-related.

Ok, we discuss MS file systems here, let’s try something more
MS-specific.

I opened MSDN online, and I entered “memory mapped files”.
The very first link found seems to be dedicated to it.
I clicked to that link, and there’s a few notes about an
“Managing Memory Mapped Files in Win32” article.
However, no link. What to do now ?

I looked at the left side of the page, where is a thing called “table of
contents”.
Suprisingly, there’s a direct link to the above mentioned article.

I do apologize once again for such written post, but this is a forum about
file systems, not about how to use google.

Good luck in writing whatever you intent.

L.

3

Thanks!!
But actually i wanted it from a filter driver’s perspective!
Where can i monitor the writes performed on those files!! What are the requests specific to these files that are seen by the filter drivers…

4

>Where can i monitor the writes performed on those files!! What are the

requests specific to these files that are seen by the filter drivers…

The write is performed to the physical pages( page frames ), and then data
from the physical pages is flushed to a disk.
To flush a page frames to a disk an IRP_MJ_WRITE request with IRP_PAGING_IO
flag is sent.

–
Slava Imameyev, xxxxx@hotmail.com

wrote in message news:xxxxx@ntfsd…
> Thanks!!
> But actually i wanted it from a filter driver’s perspective!
> Where can i monitor the writes performed on those files!! What are the
> requests specific to these files that are seen by the filter drivers…
>

5

Thanks Slava!
Can i issue a request to read the contents of that file (starting at specified offset and of specified length) when i receive an IRP_MJ_WRITE with IRP_PAGING_IO flag set??
Is it safe to issue write request on a paging io path??

6

>Can i issue a request to read the contents of that file (starting at

specified offset and of specified length) when i receive an IRP_MJ_WRITE
with IRP_PAGING_IO flag set??

Yes, but read the “Why my IRP_MJ_READ fails” message( see one message
below ).

Is it safe to issue write request on a paging io path??

It is safe, but with some limitations.

–
Slava Imameyev, xxxxx@hotmail.com

wrote in message news:xxxxx@ntfsd…
> Thanks Slava!
> Can i issue a request to read the contents of that file (starting at
> specified offset and of specified length) when i receive an IRP_MJ_WRITE
> with IRP_PAGING_IO flag set??
> Is it safe to issue write request on a paging io path??
>

7

When memory-mapped file is created, the file size query is sent to the FSD.

IO on a memory mapped file - to bring the pages in on page faults, and to
write the dirty pages - is done using the usual IRP path to the FSD, with the
IRP_NOCACHE and IRP_xxx_PAGING_IO flags (xxx can be none of SYNCHRONOUS - which
means - you can use KeWaitForSingleObject in this path, otherwise, you must do
IoMarkIrpPending/send the IRP to a work item/return STATUS_PENDING. For
non-paging IRPs, this is governed by the file object flag, for paging ones - by
the IRP itself).

Such IO is MDL-based, and Irp->MdlAddress point to a very special MDL built
by Mm internals without using IoAllocateMdl, and built as a tail of a larger
structure called “inpage support block” (for reads). So, never ever pass the
MDLs from paging IO IRPs to IoFreeMdl.

If the file is also opened the usual way with caching - then the same set
of physical pages is re-used for both the cache and the MMF implementation.
They are coherent in current Windows architecture (though Win32 docs says about
no coherency guarantees).

FSD has no other outcalls for MMFs except the file size query and paging IO
IRPs. Nothing else, though IIRC there was a FO_USER_MAPPED_FILE flag in the
FileObject.

To support MMFs, the FSD must:

  • have an FCB. This is a large structure, its format is FSD-specific, and
    the on-disk file can only have 0 or 1 in-memory FCBs - if many apps open the
    file, then 1 FCB is reused for all of them. So, many file objects can reference
    1 FCB, and FCB must be reference-counted - killed at last MJ_CLOSE for a last
    file referencing it.
  • create the FSRTL_COMMON_FCB_HEADER (or its advanced version) somewhere in
    the FCB, this structure contains 2 ERESOURCE locks (main and paging) and the
    file sizes.
    When the MJ_CREATE path sets up FileObject, the ->FsContext field must
    point to this structure within FCB. This is also a way for all other paths to
    find the FCB from file object.
  • create the SECTION_OBJECT_POINTERS structure somewhere in the FCB, it is
    3 PVOID pointers (Cc and Mm will fill them with pointers to their proprietary
    structures), must be initialized to 3 NULLs.
    In MJ_CREATE, FileObject->SectionObjectPointer must point to this FCB
    location.
  • ensure you’re responding properly to file size queries and to “set file
    size” path.
  • be prepared to receive paging IO IRPs, and remember the above-mentioned
    detail about their MDLs.
  • paging IO IRPs can continue to arrive even after MJ_CLEANUP, till the
    very MJ_CLOSE.
  • about ERESOURCE locks in the FCB header. Any file truncation must grab
    them both exclusively, first Main, then Paging. The paging IO IRPs only grab
    Paging lock shared, non-paging IO grabs Main lock shared. Any file growth grabs
    Main lock exclusively. This allows Cc/Mm to go on flushing dirty pages in the
    beginning of the file while the file is growing.
  • provide the lock acquisition FastIo routines. In these routines, grab the
    proper FCB lock and set the value via IoSetTopLevelrp which will denote in what
    way the locks were grabbed.
  • before grabbing the locks in IRP read/write path, look at
    IoGetTopLevelIrp and, if it non-NULL, then some locks were already grabbed by
    FastIo callback, behave accordingly.
  • AcquireForCreateSection callback is called by Mm/Cc when they want to
    update the 3 PVOIDs in SECTION_OBJECT_POINTERS. Grab both locks exclusively,
    first Main, then Paging.

FastFat is a good sample on all of this.

That’s all about the memory-mapped files from the FSD’s perspective. Not to
say that, to support Cc’s caching, you must support all of the above + some
additional Cc-specific things.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

8

> the on-disk file can only have 0 or 1 in-memory FCBs - if many apps open

the

The on disk file may have the data streams and each data stream has its FCB.

  • about ERESOURCE locks in the FCB header. Any file truncation must
    grab
    them both exclusively, first Main, then Paging. The paging IO IRPs only
    grab
    Paging lock shared, non-paging IO grabs Main lock shared.

FSDs don’t have to use these ERESOURCEs, they can use other techniques.
The system never uses these resources if callbacks are provided and these
callbacks return STATUS_SUCCESS or STATUS_CANT_WAIT.
Using these ERESOURCEs instead callbacks is an old NT locking model and it
depends from the flags in the FCB( ACQUIRE_MAIN_… ).

This allows Cc/Mm to go on flushing dirty pages in the beginning of the
file while the file is growing.

This allows to maintain a lock hierarchy between the Memory Manager, the
Cache Manager and FSDs.

  • provide the lock acquisition FastIo routines. In these routines …
    set the value via IoSetTopLevelrp which will denote in what
    way the locks were grabbed.

It must be pointed out that the system sets predefined top level value for
the Mapped and Modified Page Writer threads.

–
Slava Imameyev, xxxxx@hotmail.com

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntfsd…
> When memory-mapped file is created, the file size query is sent to the
> FSD.
>
> IO on a memory mapped file - to bring the pages in on page faults, and
> to
> write the dirty pages - is done using the usual IRP path to the FSD, with
> the
> IRP_NOCACHE and IRP_xxx_PAGING_IO flags (xxx can be none of SYNCHRONOUS -
> which
> means - you can use KeWaitForSingleObject in this path, otherwise, you
> must do
> IoMarkIrpPending/send the IRP to a work item/return STATUS_PENDING. For
> non-paging IRPs, this is governed by the file object flag, for paging
> ones - by
> the IRP itself).
>
> Such IO is MDL-based, and Irp->MdlAddress point to a very special MDL
> built
> by Mm internals without using IoAllocateMdl, and built as a tail of a
> larger
> structure called “inpage support block” (for reads). So, never ever pass
> the
> MDLs from paging IO IRPs to IoFreeMdl.
>
> If the file is also opened the usual way with caching - then the same
> set
> of physical pages is re-used for both the cache and the MMF
> implementation.
> They are coherent in current Windows architecture (though Win32 docs says
> about
> no coherency guarantees).
>
> FSD has no other outcalls for MMFs except the file size query and
> paging IO
> IRPs. Nothing else, though IIRC there was a FO_USER_MAPPED_FILE flag in
> the
> FileObject.
>
> To support MMFs, the FSD must:
> - have an FCB. This is a large structure, its format is FSD-specific,
> and
> the on-disk file can only have 0 or 1 in-memory FCBs - if many apps open
> the
> file, then 1 FCB is reused for all of them. So, many file objects can
> reference
> 1 FCB, and FCB must be reference-counted - killed at last MJ_CLOSE for a
> last
> file referencing it.
> - create the FSRTL_COMMON_FCB_HEADER (or its advanced version)
> somewhere in
> the FCB, this structure contains 2 ERESOURCE locks (main and paging) and
> the
> file sizes.
> When the MJ_CREATE path sets up FileObject, the ->FsContext field must
> point to this structure within FCB. This is also a way for all other paths
> to
> find the FCB from file object.
> - create the SECTION_OBJECT_POINTERS structure somewhere in the FCB, it
> is
> 3 PVOID pointers (Cc and Mm will fill them with pointers to their
> proprietary
> structures), must be initialized to 3 NULLs.
> In MJ_CREATE, FileObject->SectionObjectPointer must point to this FCB
> location.
> - ensure you’re responding properly to file size queries and to “set
> file
> size” path.
> - be prepared to receive paging IO IRPs, and remember the
> above-mentioned
> detail about their MDLs.
> - paging IO IRPs can continue to arrive even after MJ_CLEANUP, till the
> very MJ_CLOSE.
> - about ERESOURCE locks in the FCB header. Any file truncation must
> grab
> them both exclusively, first Main, then Paging. The paging IO IRPs only
> grab
> Paging lock shared, non-paging IO grabs Main lock shared. Any file growth
> grabs
> Main lock exclusively. This allows Cc/Mm to go on flushing dirty pages in
> the
> beginning of the file while the file is growing.
> - provide the lock acquisition FastIo routines. In these routines, grab
> the
> proper FCB lock and set the value via IoSetTopLevelrp which will denote in
> what
> way the locks were grabbed.
> - before grabbing the locks in IRP read/write path, look at
> IoGetTopLevelIrp and, if it non-NULL, then some locks were already grabbed
> by
> FastIo callback, behave accordingly.
> - AcquireForCreateSection callback is called by Mm/Cc when they want to
> update the 3 PVOIDs in SECTION_OBJECT_POINTERS. Grab both locks
> exclusively,
> first Main, then Paging.
>
> FastFat is a good sample on all of this.
>
> That’s all about the memory-mapped files from the FSD’s perspective.
> Not to
> say that, to support Cc’s caching, you must support all of the above +
> some
> additional Cc-specific things.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>

9

In mini filter level at IRP_MJ_CREATERE irp call in PreOperation call back I’m receiving FsContext is NULL.
It’s possible to set at post call back Operation.

If possible send any sample code, in minifilter level

Thanks & Reagrds,
Lokanadham R