File Mapping

[mani_singh]

Hi All,

Just want to know the moment when the file is mapped into the memory and when gets unmapped. is there any way to know???

Memory manager is not flushing the file ref, for those files my minifilter processed.

any pointers?

Thanks.

[rod_widdowson]

> Just want to know the moment when the file is mapped into the memory and

when gets unmapped. is there any way to know???

For mapping, as a miniFilter you will get the
IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHRONIZATION callback.

There are no callbacks for unmapping. Often you will see a MJ_CLOSE on the
file object.

Memory manager is not flushing the file ref, for those files my minifilter
processed.

Memory Manage will only do that when it needs to. Waiting days (or even
weeks) is not unknown. There are programs out there which put the system
under memory pressure, but as a minifilter you just need to let nature take
its course.

I’d start by convincing myself that Memory Manage really isn’t unmapping the
file (by doing the memory pressure trick). If that’s the case its a very
good bet that it isn’t Memory Manager but something in your code. What are
the symptoms you are seeing?

Rod

[dhananjay_kr]

Hi Rod,

You say that “for unmapping. Often you will see an IRP_MJ_CLOSE on the file
object”. My understanding is that an
IRP_MJ_CLOSE is totally unrelated to unmapping a file. You can close the
handle to the file object that you used to create the section object. Doing
mapping/unmapping sometime later. In that case assuming that this file
object has no othere references wouldn’t a CLOSE be totally unrelated or am
I missing something?

Thanks

On Wed, Mar 18, 2015 at 5:06 PM, Rod Widdowson
wrote:

> Just want to know the moment when the file is mapped into the memory and
>> when gets unmapped. is there any way to know???
>>
>
> For mapping, as a miniFilter you will get the IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHRONIZATION
> callback.
>
> There are no callbacks for unmapping. Often you will see a MJ_CLOSE on
> the file object.
>
> Memory manager is not flushing the file ref, for those files my
>> minifilter processed.
>>
>
> Memory Manage will only do that when it needs to. Waiting days (or even
> weeks) is not unknown. There are programs out there which put the system
> under memory pressure, but as a minifilter you just need to let nature take
> its course.
>
> I’d start by convincing myself that Memory Manage really isn’t unmapping
> the file (by doing the memory pressure trick). If that’s the case its a
> very good bet that it isn’t Memory Manager but something in your code.
> What are the symptoms you are seeing?
>
> Rod
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[OSR_Community_User]

You say that “for unmapping. Often you will see an IRP_MJ_CLOSE on the file object”. My understanding is that an
IRP_MJ_CLOSE is totally unrelated to unmapping a file. You can close the handle to the file object that you used to create the section object. Doing mapping/unmapping sometime later. In that case assuming that this file object has no othere references wouldn’t a CLOSE be totally unrelated or am I missing something?

The section object the Memory Manager uses maintains a reference on a specific FILE_OBJECT to ensure that it doesn’t go away. It also uses this FILE_OBJECT for paging I/O. Mm keeps that reference until it doesn’t need it anymore - like when it no longer has any data cached in memory. At that point, it calls ObDereferenceObject, which then calls IopDeleteFile, which issues the IRP_MJ_CLOSE call.

So Rod’s comment is based upon an understanding of common behavior, but is not a requirement. His initial statement (that you do not get any calls when the file is unmapped) is true.

There is a function that will tell you if the file is mapped for WRITE: MmDoesFileHaveUserWritableReferences. It doesn’t guarantee there aren’t read-only mappings of a file, but it does tell you if there are writable mappings.

Tony
OSR

[dhananjay_kr]

The section object the Memory Manager uses maintains a reference on a
specific FILE_OBJECT to ensure that it doesn’t go away. It also uses this
FILE_OBJECT for paging I/O. Mm keeps that reference until it doesn’t need
it anymore - like when it no longer has any data cached in memory. At that
point, it calls ObDereferenceObject, which then calls IopDeleteFile, which
issues the IRP_MJ_CLOSE call.

We will see an IRP_MJ_CLOSE on the backing file object when the Memory
Manager releases it’s reference. But there may be many section objects of
the same file and mappings/unmappings happening in the mean time. We won’t
be receiving any IRP_MJ_CLOSEs for those unmappings. Only when there are no
more SECTION_OBJECTs for the file and the Memory Manager decides it no
longer needs the control area it will release it’s reference to the backing
file object. We will see an IRP_MJ_CLOSE then. So for ‘n’ unmappings of the
file you may see only one IRP_MJ_CLOSE and we are not even sure when that
IRP_MJ_CLOSE will come?

Is the above understanding correct?

Thanks

On Wed, Mar 18, 2015 at 8:13 PM, Tony Mason wrote:

>

> You say that “for unmapping. Often you will see an IRP_MJ_CLOSE on the
> file object”. My understanding is that an
> IRP_MJ_CLOSE is totally unrelated to unmapping a file. You can close the
> handle to the file object that you used to create the section object. Doing
> mapping/unmapping sometime later. In that case assuming that this file
> object has no othere references wouldn’t a CLOSE be totally unrelated or am
> I missing something?
>

>
> The section object the Memory Manager uses maintains a reference on a
> specific FILE_OBJECT to ensure that it doesn’t go away. It also uses this
> FILE_OBJECT for paging I/O. Mm keeps that reference until it doesn’t need
> it anymore - like when it no longer has any data cached in memory. At that
> point, it calls ObDereferenceObject, which then calls IopDeleteFile, which
> issues the IRP_MJ_CLOSE call.
>
> So Rod’s comment is based upon an understanding of common behavior, but is
> not a requirement. His initial statement (that you do not get any calls
> when the file is unmapped) is true.
>
> There is a function that will tell you if the file is mapped for WRITE:
> MmDoesFileHaveUserWritableReferences. It doesn’t guarantee there aren’t
> read-only mappings of a file, but it does tell you if there are writable
> mappings.
>
> Tony
> OSR
>
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[OSR_Community_User]

We will see an IRP_MJ_CLOSE on the backing file object when the Memory Manager releases it’s reference. But there may be many section objects of the same file and mappings/unmappings happening in the mean time. We won’t be receiving any IRP_MJ_CLOSEs for those unmappings. Only when there are no more SECTION_OBJECTs for the file and the Memory Manager decides it no longer needs the control area it will release it’s reference to the backing file object. We will see an IRP_MJ_CLOSE then. So for ‘n’ unmappings of the file you may see only one IRP_MJ_CLOSE and we are not even sure when that IRP_MJ_CLOSE will come?

There can only be *at most* two section objects for a file: one for executing it (as a program) and one for accessing it (as data). There may be multiple *mappings* to that section (regions in a virtual address space that are “backed by” the section object). You won’t receive notification when someone creates or tears down a mapping.

The IRP_MJ_CLOSE comes in when the reference count on the file object drops to zero. While that often happens when the section is torn down, it’s not guaranteed.

So, bottom line, what you want (callbacks from Mm when mappings are created and torn down) does not exist in Windows.

Tony
OSR

[dhananjay_kr]

Thanks a lot for the explanation.

On Wed, Mar 18, 2015 at 9:42 PM, Tony Mason wrote:

>

> We will see an IRP_MJ_CLOSE on the backing file object when the Memory
> Manager releases it’s reference. But there may be many section objects of
> the same file and mappings/unmappings happening in the mean time. We won’t
> be receiving any IRP_MJ_CLOSEs for those unmappings. Only when there are no
> more SECTION_OBJECTs for the file and the Memory Manager decides it no
> longer needs the control area it will release it’s reference to the backing
> file object. We will see an IRP_MJ_CLOSE then. So for ‘n’ unmappings of the
> file you may see only one IRP_MJ_CLOSE and we are not even sure when that
> IRP_MJ_CLOSE will come?
>

>
> There can only be at most two section objects for a file: one for
> executing it (as a program) and one for accessing it (as data). There may
> be multiple mappings to that section (regions in a virtual address space
> that are “backed by” the section object). You won’t receive notification
> when someone creates or tears down a mapping.
>
> The IRP_MJ_CLOSE comes in when the reference count on the file object
> drops to zero. While that often happens when the section is torn down, it’s
> not guaranteed.
>
> So, bottom line, what you want (callbacks from Mm when mappings are
> created and torn down) does not exist in Windows.
>
> Tony
> OSR
>
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[Pavel_Lebedinsky]

The terms “section” and “control area” are often used interchangeably, but technically it’s the control areas that are limited to two per file (one for executable image sections, and one for data sections). Each control area can have 0 to N section objects pointing to it, and 0 to N mapped views. Multiple CreateFileMapping calls on the same file will create multiple section objects, but they will all point to the same control area.

lkd> !fileobj ffffe000c0986e10

\Users\test\t.tmp

Section Object Pointers: ffffe000bbaaccb8

lkd> dt nt!_SECTION_OBJECT_POINTERS ffffe000bbaaccb8
+0x000 DataSectionObject : 0xffffe000`c0a5a8a0 Void <- really a control area pointer
+0x008 SharedCacheMap : (null)
+0x010 ImageSectionObject : (null)

lkd> !ca 0xffffe000`c0a5a8a0

ControlArea @ ffffe000c0a5a8a0
Segment ffffc002137d0480 Flink ffffe000b3b259a0 Blink ffffe000b3b259a0
Section Ref 1 Pfn Ref 40 Mapped Views 1
User Ref 2 WaitForDel 0 Flush Count 0
File Object ffffe000c0986e10 ModWriteCount 0 System Views 0
WritableRefs 2
Flags (8080) File WasPurged

\Users\test\t.tmp

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Wednesday, March 18, 2015 9:13 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] File Mapping

We will see an IRP_MJ_CLOSE on the backing file object when the Memory Manager releases it’s reference. But there may be many section objects of the same file and mappings/unmappings happening in the mean time. We won’t be receiving any IRP_MJ_CLOSEs for those unmappings. Only when there are no more SECTION_OBJECTs for the file and the Memory Manager decides it no longer needs the control area it will release it’s reference to the backing file object. We will see an IRP_MJ_CLOSE then. So for ‘n’ unmappings of the file you may see only one IRP_MJ_CLOSE and we are not even sure when that IRP_MJ_CLOSE will come?

There can only be *at most* two section objects for a file: one for executing it (as a program) and one for accessing it (as data). There may be multiple *mappings* to that section (regions in a virtual address space that are “backed by” the section object). You won’t receive notification when someone creates or tears down a mapping.

The IRP_MJ_CLOSE comes in when the reference count on the file object drops to zero. While that often happens when the section is torn down, it’s not guaranteed.

So, bottom line, what you want (callbacks from Mm when mappings are created and torn down) does not exist in Windows.

Tony
OSR

---

NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[OSR_Community_User]

The terms “section” and “control area” are often used interchangeably, but technically it’s the control areas that are limited to two per file (one for executable image sections, and one for data sections). Each control area can have 0 to N section objects pointing to it, and 0 to N mapped views. Multiple CreateFileMapping calls on the same file will create multiple section objects, but they will all point to the same control area.

In general, I’ve eschewed describing the distinction because “control area” is something that is internal to the specific implementation (though I do point out that one can use the !ca debugger command on the DataSectionObject and ImageSectionObject) and is not a documented concept/abstraction. And, as you observe, Mm still only has one pointer to the file object from the control area (with two possible control areas for different types of access). But I do routinely explain to people that an extra level of indirection is the norm in the OS business, so one extra indirection shouldn’t surprise anyone.

Of course, as far as the OP is concerned, he still isn’t going to get a callback for mapping, section OR control area teardown.

Tony
OSR

[OSR_Community_User]

>IRP_MJ_CLOSE then. So for ‘n’ unmappings of the file you may see only one IRP_MJ_CLOSE and

we are not even sure when that IRP_MJ_CLOSE will come?

Yes.

Section object handle references the ‘Sect’ structure, which contains 2 fields: a) allowed page protection flags b) ref to “control area” ‘MmCa’ object.

MmCa is always 1 per on-disk file and is 1-to-1 with the FCB. Well, either 0 (if not created yet) or 1.

MmCa is what is used by Mm to implement the mappings. And yes, it holds a ref to some FO which will be used for paging IO.

Also, MmCa is used by Cc.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

[OSR_Community_User]

> There can only be *at most* two section objects for a file: one for executing it (as a program) and

one for accessing it (as data).

This is true on “control areas” (MmCa/MmCi) but not sections.

Call CreateFileMapping 2 times, especially with different protection flags - and look at 2 sections objects for the same file.

MmCa is still the same, surely.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

[OSR_Community_User]

There can only be *at most* two section objects for a file: one for
executing it (as a program) and one for accessing it (as data).

This is true on “control areas” (MmCa/MmCi) but not sections.

From the file systems level, the MmCa IS the section - it’s what SOP points to (DataSectionObject or ImageSectionObject). The “control area” is not a documented structure or concept in the WDK.

Tony
OSR

[OSR_Community_User]

(Tony, I know for sure you know all of this better then me, but let’s write this for the sake of forum archives).

From the file systems level, the MmCa IS the section - it’s what SOP points to (DataSectionObject
or ImageSectionObject).

Exactly.

More so, this is a weak pointer (does not hold a ref).

This is since MmCa has a strong (ref-holding) pointer to some File, and File definitely has a strong pointer to its FCB (FileObject->FsContext), which in turns contain SectionObjectPointers.

This is even more interesting when Cc comes to play. Shared cache map (CcSc) holds a strong ref to MmCa, and, in turn, the lifetime of CcSc, after the last file called CcUninitializeCacheMap, is determined by the Cc’s garbage collector which can allow CcSc to survive for hours if not days.

Then you have the ref chain of CcSc -> MmCa -> File, and thus no MJ_CLOSE till the garbage collector wants to destroy CcSc.

And yes, at least in older Windows the MmSt “segment” prototype PTE tables were owned by MmCa. In turn, MmSt entries were “owning” (in some logical sense of view) the physical pages mapped to this file.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

[OSR_Community_User]

From the file systems level, the MmCa IS the section - it’s what SOP
points to (DataSectionObject or ImageSectionObject).

Exactly.

More so, this is a weak pointer (does not hold a ref).

This is since MmCa has a strong (ref-holding) pointer to some File, and File definitely has a strong pointer to its FCB (FileObject->FsContext), which in turns contain SectionObjectPointers.

This is even more interesting when Cc comes to play. Shared cache map (CcSc) holds a strong ref to MmCa, and, in turn, the lifetime of CcSc, after the last file called CcUninitializeCacheMap, is determined by the Cc’s garbage collector which can allow CcSc to survive for hours if not days.

Then you have the ref chain of CcSc -> MmCa -> File, and thus no MJ_CLOSE till the garbage collector wants to destroy CcSc.

And yes, at least in older Windows the MmSt “segment” prototype PTE tables were owned by MmCa. In turn, MmSt entries were “owning” (in some logical sense of view) the physical pages mapped to this file.

And *that* is why I’ve always just taught the conceptual model that the control area is a section. The implementation details really have (and continue) to change. Trying to give people “too much” detail (like during the file systems class) is actually harmful. By the time they get to where they understand this level of conversation (oh, that “section object” that the FSD sees indirectly via FO->SOP is really called a “control area” and the “section object” is yet another data structure that points to it…) it doesn’t screw up the model with too much implementation detail.

I have a picture I use in class to describe this interaction (http://tinyurl.com/ozrd5bk). It’s already fairly complex and from a conceptual framework it doesn’t anything if we add the control area concept.

Tony
OSR

[mani_singh]

While doing MmForceSectionClosed() doesn’t helping me since mm is still having some ref
to file object.

i think i need to stick @ rod: but as a minifilter you just need to let nature take
its course.

Nice depth knowledge sharing thanks a lot all.

[mani_singh]

One more question is there any way to put memory pressure on standby list as a minifilter.

I think there is no memory pressure so the pages are remain on the standby list, easily seen by tool.

any pointers??

Thanks a lot.

[dhananjay_kr]

Hi Mani,

Can you please explain what are you trying to achieve by moving pages from
the standby list to the free list?

Thanks

On Tue, Mar 24, 2015 at 10:57 AM, wrote:

> One more question is there any way to put memory pressure on standby list
> as a minifilter.
>
> I think there is no memory pressure so the pages are remain on the standby
> list, easily seen by tool.
>
> any pointers??
>
> Thanks a lot.
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[OSR_Community_User]

Inject artificial extra page faults probably?

“Dhananjay Kumar” wrote in message news:xxxxx@ntfsd…
Hi Mani,

Can you please explain what are you trying to achieve by moving pages from the standby list to the free list?

Thanks

On Tue, Mar 24, 2015 at 10:57 AM, wrote:

One more question is there any way to put memory pressure on standby list as a minifilter.

I think there is no memory pressure so the pages are remain on the standby list, easily seen by tool.

any pointers??

Thanks a lot.

—
NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[OSR_Community_User]

I think at least some Driver Verifier options were doing this.

Read the DV docs.

wrote in message news:xxxxx@ntfsd…
> One more question is there any way to put memory pressure on standby list as a minifilter.
>
> I think there is no memory pressure so the pages are remain on the standby list, easily seen by tool.
>
> any pointers??
>
> Thanks a lot.
>