Separate cache for images

Is there a separate cache for loaded images? In my isolation filter, I noticed that non-Windows images when loading into a process will not trigger a IRP_MJ_READ more than once. I still see reads for Windows system files (ntdll, kernel32) but the non Windows don’t seem to cause a read even though I can see the image load during the callback. So either my isolation filter is not working properly or there there is a separate cache. And if there is a separate cache, is there a way to force a read to occur? That’s why I wrote an isolation filter to begin with.

> Is there a separate cache for loaded images?

As in executable images? Yes. Check out the FAT sources and look for
MmFlushImageSection

The system keeps physical pages for an executable image section in the active or standby lists even if no process maps the image file. Pages are kept resident with valid data until the number of free pages in the system drops below some threshold and the Memory Manager reclaims pages.

When a CPU accesses a virtual address range backed by an executable image section a page fault handler doesn’t call a file system to read page from a file if a related prototype page table entry has a valid page reference. The page fault handler establishes a virtual to physical mapping for this physical page and increments a shared count for the physical page.

That said, you might not see read requests because pages are in the active or standby lists as a result of access in the past OR because an executable image code has never been executed or accessed. Mapping an executable image doesn’t equate to its execution.

This requires invalidating all virtual to physical mappings in all processes that map this page and this functionality is not exported. In normal operating mode this happens when the Working Set Manager trims process working set in response to the free pages pool/list depletion.

Drivers can use MmFlushImageSection which invalidates prototype page table entries if there is no mapped view for an image section.

Thanks Rod and Slava.

I was already using MmFlushImageSection in CREATE and SET_INFO (delete and rename) but wasn’t using it in CLEANUP like the docs mention. So I added it there however, whenever I call it from CLEANUP it always returns FALSE for the non system files. Some system files return TRUE while others also always return FALSE.

In CLEANUP, should MmFlush be called before or after Cc functions or does it not matter? I get the same results either way but just curious on the proper flow of things. And do all handles have to be closed when calling MmFlush or will the memory manager mark it as flushable and do its thing once the ref count reaches zero?

So when MmFlush fails, that means there is still a mapped view of the section. I find it unusual that ntdll will return TRUE but every process running will have a mapped view of that image. Any thoughts on where a reference is being missed (I’m assuming it’s a reference thing) for Mm to think the image is still mapped?

Before. It is safer.

File handles are not related to section control area in this way. MmFlushImageSection checks for section control area references( i.e. checks for NtCreateSection) and mapped views ( i.e. NtMapViewOfSection).

Below is a WinDBG output for ntdll.dll control area for Win10 16299.15.amd64fre.rs3_release.170928-1534 . The number of mapped views and user references are not zero. I don’t have a reasonable explanation for MmFlushImageSection returning TRUE for this control area.

3: kd> !ca 0xffffb70b`8eb78790

ControlArea @ ffffb70b8eb78790
Segment ffffd983a2b485a0 Flink ffffb70b8ff86110 Blink ffffb70b8dbfb470
Section Ref 2 Pfn Ref 169 Mapped Views 3d
User Ref 3f WaitForDel 0 Flush Count 8a40
File Object ffffb70b8e7ccb70 ModWriteCount 0 System Views 7264
WritableRefs c0001e
Flags (a0) Image File

\Windows\System32\ntdll.dll

Segment @ ffffd983a2b485a0
ControlArea ffffb70b8eb78790 BasedAddress 00007ff98d7c0000
Total Ptes 1e0
Segment Size 1e0000 Committed 0
Image Commit c Image Info ffffd983a2b485e8
ProtoPtes ffffd983a2b79010
Flags (c2822000) DebugSymbolsLoaded ProtectionMask

Subsection 1 @ ffffb70b8eb78810
ControlArea ffffb70b8eb78790 Starting Sector 0 Number Of Sectors 2
Base Pte ffffd983a2b79010 Ptes In Subsect 1 Unused Ptes 0
Flags 2 Sector Offset 0 Protection 1

Subsection 2 @ ffffb70b8eb78848
ControlArea ffffb70b8eb78790 Starting Sector 2 Number Of Sectors 887
Base Pte ffffd983a2b79018 Ptes In Subsect 111 Unused Ptes 0
Flags 6 Sector Offset 0 Protection 3

Subsection 3 @ ffffb70b8eb78880
ControlArea ffffb70b8eb78790 Starting Sector 889 Number Of Sectors 1
Base Pte ffffd983a2b798a0 Ptes In Subsect 1 Unused Ptes 0
Flags 6 Sector Offset 0 Protection 3

Subsection 4 @ ffffb70b8eb788b8
ControlArea ffffb70b8eb78790 Starting Sector 88a Number Of Sectors 22c
Base Pte ffffd983a2b798a8 Ptes In Subsect 46 Unused Ptes 0
Flags 2 Sector Offset 0 Protection 1

Subsection 5 @ ffffb70b8eb788f0
ControlArea ffffb70b8eb78790 Starting Sector ab6 Number Of Sectors 20
Base Pte ffffd983a2b79ad8 Ptes In Subsect 8 Unused Ptes 0
Flags a Sector Offset 0 Protection 5

Subsection 6 @ ffffb70b8eb78928
ControlArea ffffb70b8eb78790 Starting Sector ad6 Number Of Sectors 70
Base Pte ffffd983a2b79b18 Ptes In Subsect e Unused Ptes 0
Flags 2 Sector Offset 0 Protection 1

Subsection 7 @ ffffb70b8eb78960
ControlArea ffffb70b8eb78790 Starting Sector b46 Number Of Sectors 1b
Base Pte ffffd983a2b79b88 Ptes In Subsect 4 Unused Ptes 0
Flags a Sector Offset 0 Protection 5

Subsection 8 @ ffffb70b8eb78998
ControlArea ffffb70b8eb78790 Starting Sector b61 Number Of Sectors 1
Base Pte ffffd983a2b79ba8 Ptes In Subsect 1 Unused Ptes 0
Flags 2 Sector Offset 0 Protection 1

Subsection 9 @ ffffb70b8eb789d0
ControlArea ffffb70b8eb78790 Starting Sector b62 Number Of Sectors 351
Base Pte ffffd983a2b79bb0 Ptes In Subsect 6b Unused Ptes 0
Flags 2 Sector Offset 0 Protection 1

Subsection 10 @ ffffb70b8eb78a08
ControlArea ffffb70b8eb78790 Starting Sector eb3 Number Of Sectors 3
Base Pte ffffd983a2b79f08 Ptes In Subsect 1 Unused Ptes 0
Flags 2 Sector Offset 0 Protection 1

I’ve tried calling MmFlushImageSection in my driver to no avail. It doesn’t seem to have any effect on the image loading behavior.

Is there a way to lower this threshold or temporarily cross it so it causes a purge?

Again, any way to force this condition through other means?

Shouldn’t there be at least one read initially?

I wrote a test case where the parent app copies an existing dll to a new directory. In that directory there’s another application that will call LoadLibrary on the new dll (different filename than the original). However as I monitor in ProcMon, there are no IRP_MJ_READ requests on the new dll. It gets written, ACQUIRED_SECTION_SYNC is called a few times and then the load image callback is called. The output is below.

Looking at FastFat, there are certain conditions where it calls MmFlushImageSection (and I assume it’s similar with NTFS) and I tried to recreate those situations in my user mode app (you’ll see a SetDispositionInformationFile toggle on the new file in the ProcMon output below). Still doesn’t seem to help.

I guess I’ll start from the beginning so maybe someone smarter than me can suggest something. I have a requirement where a DLL gets injected into certain applications. This DLL has some place holders in a section that needs to get filled in before execution begins. That is something I can’t change. How I modify the placeholders is under my control. The easiest would be if I could just modify the data during PsSetLoadImageNotifyRoutine but page protections can’t be changed because of the address space lock. Modifying on read seemed like a good option, however with the image caching that doesn’t seem to play by normal rules (at least in my Win7 VM) I’m not sure if that’s a viable solution either. It doesn’t have to be done in the kernel as I can pass the state information to user-mode but there may be an issue with execution (can I modify in user-mode before DllAttach is called?) Any thoughts on the best way to accomplish this would be helpful.

Parent.exe 3056 IRP_MJ_CREATE C:\Program Files\Test\temp\dll32.dll SUCCESS Desired Access: Generic Write, Read Data/List Directory, Read Attributes, Delete, Disposition: Create, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: ANCI, ShareMode: None, AllocationSize: 0, Impersonating: NT AUTHORITY\SYSTEM, OpenResult: Created
Parent.exe 3056 IRP_MJ_QUERY_VOLUME_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryAttributeInformationVolume, FileSystemAttributes: Case Preserved, Case Sensitive, Unicode, ACLs, Compression, Named Streams, EFS, Object IDs, Reparse Points, Sparse Files, Quotas, Transactions, 0x3c00000, MaximumComponentNameLength: 255, FileSystemName: NTFS
Parent.exe 3056 FASTIO_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryBasicInformationFile, CreationTime: 11/14/2017 8:24:17 PM, LastAccessTime: 11/14/2017 8:24:17 PM, LastWriteTime: 11/14/2017 8:24:17 PM, ChangeTime: 11/14/2017 8:24:17 PM, FileAttributes: A
Parent.exe 3056 IRP_MJ_SET_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: SetEndOfFileInformationFile, EndOfFile: 42,760
Parent.exe 3056 IRP_MJ_WRITE C:\Program Files\Test\temp\dll32.dll SUCCESS Offset: 0, Length: 42,760, Priority: Normal
Parent.exe 3056 IRP_MJ_SET_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: SetBasicInformationFile, CreationTime: 0, LastAccessTime: 0, LastWriteTime: 11/8/2017 12:38:09 PM, ChangeTime: 11/14/2017 8:23:48 PM, FileAttributes: n/a
Parent.exe 3056 IRP_MJ_CLEANUP C:\Program Files\Test\temp\dll32.dll SUCCESS
Parent.exe 3056 IRP_MJ_CREATE C:\Program Files\Test\temp\dll32.dll SUCCESS Desired Access: Generic Write, Read Attributes, Delete, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, Impersonating: NT AUTHORITY\SYSTEM, OpenResult: Opened
Parent.exe 3056 IRP_MJ_SET_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: SetDispositionInformationFile, Delete: True
Parent.exe 3056 IRP_MJ_SET_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: SetDispositionInformationFile, Delete: False
Parent.exe 3056 IRP_MJ_CLEANUP C:\Program Files\Test\temp\dll32.dll SUCCESS
Parent.exe 3056 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS
System 4 IRP_MJ_WRITE C:\Program Files\Test\temp\dll32.dll SUCCESS Offset: 0, Length: 45,056, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
System 4 IRP_MJ_SET_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: SetEndOfFileInformationFile, EndOfFile: 42,760
System 4 FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS SyncType: SyncTypeOther
System 4 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 FASTIO_NETWORK_QUERY_OPEN C:\Program Files\Test\temp\dll32.dll FAST IO DISALLOWED
Test.exe 3112 IRP_MJ_CREATE C:\Program Files\Test\temp\dll32.dll SUCCESS Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
Test.exe 3112 FASTIO_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryBasicInformationFile, CreationTime: 11/14/2017 8:24:17 PM, LastAccessTime: 11/14/2017 8:24:17 PM, LastWriteTime: 11/8/2017 12:38:09 PM, ChangeTime: 11/14/2017 8:23:48 PM, FileAttributes: A
Test.exe 3112 IRP_MJ_CLEANUP C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 IRP_MJ_CREATE C:\Program Files\Test\temp\dll32.dll SUCCESS Desired Access: Read Data/List Directory, Execute/Traverse, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, Delete, AllocationSize: n/a, OpenResult: Opened
Test.exe 3112 FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll FILE LOCKED WITH ONLY READERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE
Test.exe 3112 FASTIO_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryStandardInformationFile, AllocationSize: 45,056, EndOfFile: 42,760, NumberOfLinks: 1, DeletePending: False, Directory: False
Test.exe 3112 FASTIO_ACQUIRE_FOR_CC_FLUSH C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 FASTIO_RELEASE_FOR_CC_FLUSH C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS SyncType: SyncTypeOther
Test.exe 3112 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 IRP_MJ_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryNameInformationFile, Name: \Program Files\Test\temp\dll32.dll
Test.exe 3112 Load Image C:\Program Files\Test\temp\dll32.dll SUCCESS Image Base: 0x6b6b0000, Image Size: 0xf000
Test.exe 3112 IRP_MJ_CLEANUP C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 IRP_MJ_CREATE C:\Program Files\Test\temp\dll32.dll SUCCESS Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, Delete, AllocationSize: n/a, OpenResult: Opened
Test.exe 3112 FASTIO_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryBasicInformationFile, CreationTime: 11/14/2017 8:24:17 PM, LastAccessTime: 11/14/2017 8:24:17 PM, LastWriteTime: 11/8/2017 12:38:09 PM, ChangeTime: 11/14/2017 8:23:48 PM, FileAttributes: A
Test.exe 3112 IRP_MJ_CLEANUP C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 IRP_MJ_CREATE C:\Program Files\Test\temp\dll32.dll SUCCESS Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, AllocationSize: n/a, OpenResult: Opened
Procmon.exe 2744 FASTIO_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryStandardInformationFile, AllocationSize: 45,056, EndOfFile: 42,760, NumberOfLinks: 1, DeletePending: False, Directory: False
Procmon.exe 2744 FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll FILE LOCKED WITH ONLY READERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_READONLY
Procmon.exe 2744 FASTIO_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryStandardInformationFile, AllocationSize: 45,056, EndOfFile: 42,760, NumberOfLinks: 1, DeletePending: False, Directory: False
Procmon.exe 2744 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS SyncType: SyncTypeOther
Procmon.exe 2744 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 IRP_MJ_CLEANUP C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 IRP_MJ_CREATE C:\Program Files\Test\temp\dll32.dll SUCCESS Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, AllocationSize: n/a, OpenResult: Opened
Procmon.exe 2744 FASTIO_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryStandardInformationFile, AllocationSize: 45,056, EndOfFile: 42,760, NumberOfLinks: 1, DeletePending: False, Directory: False
Procmon.exe 2744 FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll FILE LOCKED WITH ONLY READERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_READONLY
Procmon.exe 2744 FASTIO_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryStandardInformationFile, AllocationSize: 45,056, EndOfFile: 42,760, NumberOfLinks: 1, DeletePending: False, Directory: False
Procmon.exe 2744 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS SyncType: SyncTypeOther
Procmon.exe 2744 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 IRP_MJ_CLEANUP C:\Program Files\Test\temp\dll32.dll SUCCESS
Procmon.exe 2744 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS
Test.exe 3112 IRP_MJ_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryNameInformationFile, Name: \Program Files\Test\temp\dll32.dll
Parent.exe 3056 IRP_MJ_CREATE C:\Program Files\Test\temp\dll32.dll SUCCESS Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-Directory File, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
Parent.exe 3056 IRP_MJ_QUERY_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: QueryAttributeTagFile, Attributes: A, ReparseTag: 0x0
Parent.exe 3056 IRP_MJ_SET_INFORMATION C:\Program Files\Test\temp\dll32.dll SUCCESS Type: SetDispositionInformationFile, Delete: True
Parent.exe 3056 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS
Parent.exe 3056 IRP_MJ_CLEANUP C:\Program Files\Test\temp\dll32.dll SUCCESS
Parent.exe 3056 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS
System 4 FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS SyncType: SyncTypeOther
System 4 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\Program Files\Test\temp\dll32.dll SUCCESS
System 4 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS
System 4 IRP_MJ_CLOSE C:\Program Files\Test\temp\dll32.dll SUCCESS

A filter driver should see a paging read request when a CPU tries to fetch data from a virtual address for a PTE marked as invalid and when a page fault can’t be resolved with a page in the active or standby list ( this is what you call “image caching” ).

The one thing you should understand is that this page is shared by all processes that map this dll/exe and your filter doesn’t make it private for a process as happens for copy on write (COW) pages on first modification outside FSD stack. Your filter should also be prepared to modify a page multiple times as it gets evicted and reread again by the Memory Manager.

Yes. And there is more - you can make modified pages private for a process with NtProtectVirtualMemory( PAGE_WRITECOPY ) before modification like a debugger does before it sets a breakpoint. COW pages are backed by a page file so you don’t need to modify them again for the same process after they are reclaimed and read again from a page file.

There is some values in the registry. They are described in the “Windows Internals” book series. I believe they are read once at boot so this might be not what you want. Alternatively, it is possible to run a memory hog process with elevated privileges that allocates and commits a huge amount of memory in its virtual address space, but this might result in DoS for other subsystems including the kernel mode one.

“access” means “read” in this context. So, yes there should be a paging read to bring data from a file system to a page allocated by the Memory Manager to map an executable image.

LoadLibrary creates a file mapping structures to support future page faults but doesn’t allocate pages for an executable portion of a file. These pages are read when a page fault is resolved on first access or by a prefetcher thread.

In your case the file was written before being mapped as an image so it also had pages cached through a data section, but data section and image section don’t share pages. So this should not be an issue here.

You’re right they don’t share, but Mm will try to copy the data from the data section to the image section instead of going to disk (MiCopyDataPageToImagePage). So, it would be correct to not see a fault on the image section if you didn’t successfully flush/purge the data section after write.

-scott
OSR
@OSRDrivers

Thank you Slava and Scott.

I added a call to MmForceSectionClosed(SOP, TRUE) in PRE_CLEANUP and I’m now seeing the reads I would expect. I’m sure it’s not great for performance but at least for now it works. If only the address space lock wasn’t held in the image callback, it’d make life so much simpler.