Not to split hairs, but the file cache is managed by the cache manager (CC) not the memory manager (MM). Two intertwined but very different modules, owned by two different people in two entirely separate groups (well, when last I checked they were in different groups anyways – unsurprisingly, CC was owned by the file systems people).
Peter
OSR
@OSRDrivers
Peter Viscarola (OSR) wrote:
So while you may differ with how a particular policy is
implemented, think for a minute: When’s the last time
you had a beef with the reliability of MM?
Well, there was that crash in MmGetSystemRoutineAddress…
> -----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-556161-
xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Sunday, April 27, 2014 4:54 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Wonders of paged code sections.Peter Viscarola (OSR) wrote:
> So while you may differ with how a particular policy is implemented,
> think for a minute: When’s the last time you had a beef with the
> reliability of MM?Well, there was that crash in MmGetSystemRoutineAddress…
Thanks for illustrating Peter’s point. That was a lot of years ago.
Phil
Not speaking for LogRhythm
Phil Barila | Senior Software Engineer
720.881.5364 (w)
The Security Intelligence Company
A LEADER 2013 SIEM Magic Quadrant
Perfect 5-Star Rating in SC Magazine for 5 Consecutive Years
>Well, there was that crash in MmGetSystemRoutineAddress…
In all fairness, the only thing that links it to MM is the Mm* prefix.
Cc will unmap the file (putting pages on the standby list) when the app closes its handle. If the app is reading a large file sequentially Cc will also unmap each region after it’s been accessed.
So a typical file indexer kind of app should not be able to inflate the system cache to the point where the OS has to trim processes. What it can do is dump a lot of pages onto the standby list, potentially pushing other more useful pages from memory.
To avoid that, an app that wants to be a good citizen can lower its page priority via SetThreadInformation, or by entering background mode (THREAD_MODE_BACKGROUND etc). In some cases the system can do this automatically, but if you want to reduce the impact on the rest of the system it’s better to be explicit.
Using non-cached IO in scenarios like file indexing can actually make things worse because NTFS will attempt to purge cached files from memory when they are opened for non-cached access.
TL;DR:
If you need to scan a lot of files and you don’t know who else might be using them:
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@broadcom.com
Sent: Saturday, April 26, 2014 9:02 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Wonders of paged code sections.
Windows file cache, also, seems to try to keep files cached indefinitely and without ever trying to limit cache size (in the old good times, Win9x had registry-specified cache limit, though). When some crappy file scanner/indexer is opening lots of files in cached mode (or, equally bad, opens them as file sections), the executable pages seem to fall a victim of that fast.
That sounds very much like the driver bug that was just discussed here (the one that triggers bugcheck 1A/0x61946 in win8+):
http://www.osronline.com/showThread.cfm?link=255401
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@broadcom.com
Sent: Sunday, April 27, 2014 10:43 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Wonders of paged code sections.
<…>
In Win2003 (God forbid I mention it) I was very surprized to be getting “data cannot be flushed” warnings on “unsafe” unplug of a USB stick, EVEN IF NO FILE WERE MODIFIED.
>Cc will unmap the file (putting pages on the standby list) when the app closes its handle.
What if a file mapping object doesn’t have open handles to its FILE_OBJECT? Like all loaded EXEs and DLLs do? does it dis-advantage it against currently open files?
Cc doesn’t know anything about loaded EXEs and DLLs.
Cc maps views of files accessed via ReadFile/WriteFile. These views are in the system VA space, and Cc decides when to unmap them.
User programs can also map views, via MapViewFile/NtMapViewOfSection (LoadLibrary internally calls NtMapViewOfSection). These views are in the VA space of the process, and the process decides when to unmap them.
Orthogonally to all that, Mm can decide to trim active pages from a mapped view. When it decides which pages to trim first it looks at various things, but whether the file has any open handles is not one of them.
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@broadcom.com
Sent: Tuesday, April 29, 2014 4:29 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Wonders of paged code sections.
Cc will unmap the file (putting pages on the standby list) when the app closes its handle.
What if a file mapping object doesn’t have open handles to its FILE_OBJECT? Like all loaded EXEs and DLLs do? does it dis-advantage it against currently open files?