Cache Load Glitch

This is not a problem I need solved, my code handles the situation just fine, but it looks like a cache load glitch and I wanted to describe what I am seeing and get your opinion.

I am filtering cache reads (actually, all IRP_NOCACHE reads), and I’ve seen a fairly consistent pattern:

  • If you read a small data file, say 200 bytes, you get a 4K read at offset 0, which returns the entire contents of the file, and is done
  • If you read a file that is 13000 bytes, you get a 16K read at offset 0, and you’re done
  • For larger data files, in all tests I have run, you get a 32K read @ 0, then a 32K read of the last 8 pages of the file, and then a series of 32K reads to load the rest of the file, except where chunks remain that are less than 32K, which are read in 16K, 8K, or 4K chunks. Often, but not exclusively, from the start of the file to the end. Always page aligned as you would expect.
  • Large executables are slightly different, possibly due to the execution of the app forcing reads at different places, so the read is not sequential. But it still begins with reading the first and last 32K, and then performing a series of 32K reads here and there except where a remaining section is less than 32K, etc.

The irregularity I am seeing is that near the end of the load of some executables, the reads are not page aligned. One test app I have consistently loads in the same pattern that includes 9 misaligned reads at the end. The exe is 808K. It loads first and last 32K as always, and then performs 25 other reads here and there to slowly bring the entire exe into cache, mostly 32K reads, other than 3 16K, one 8K, and one 28K read.

That left 2 4K chunks at offsets 220K and 628K, and a 16K chunk at offset 664K, which it could have read with 3 page aligned reads, but instead read as 9 misaligned 4K reads as follows:

217K & 221K - to load the 4K at 220K
625K & 629K - to load the 4K at 628K
661K, 665K, 669K, 673K, and 677K - to load the 16K at 664K

What do you think of this?

_Ron

Hi,

Executable files are mapped as subsections (which are recorded in a PE header). Subsections are a set of consecutive memory ranges aligned on a page boundary. A subsection offset in a file on a disk is not required to be page aligned. This offset should be disk sector aligned (which is 0x200 == 512 by default), I presume to save disk space. You can confirm this with the !ca command looking at the “Starting Sector” value for each subsection. These values are generally not a multiple of 8 (8*0x200 == 0x1000 = 4096). When the Memory Manager brings in a page for a subsection it uses this value to calculate page offset in a file.

Anyway, I think that you didn’t provide a full picture. You were filtering IRP_NOCACHE IRPs which is not the same as IRP_PAGING_IO IRPs. IRP_NOCACHE should be disk block aligned which is only 512 bytes for HDD and SSD. You didn’t provide a call stack which could confirm this was really a paging IO from the Memory Manager to support executable file mapping.

I was filtering all IRP_NOCACHE reads, but in this case they were all IRP_NOCACHE | IRP_PAGING_IO | IRP_SYNCHRONOUS_PAGING_IO.

I was not aware that EXEs are not mapped into the cache on memory page boundaries. Seems odd that it would read everything on page boundaries but these 3 places, and then split them across pages. How would it deal with the page overlaps in the cache where parts of the file exist in two cache pages simultaneously?

_Ron

Windows is different from many UNIX system by having a separate maps for data and executable files. Files mapped as an executable image don’t share cache with data files. They do not share virtual or physical pages. I will use “cache” and “file mapping” interchangeably as the cache is supported by file mapping structures, though pages might not be mapped in any address space they are retained by a control area structure related to file mapping.
If you map a file as a data file and as an executable file there are two copies of data in the system - one in the data cache and the other as a mapped executable (you can call this “executable cache” if you want). A data related mapping is supported by FileObject->SectionObjectPointer->DataSectionObject and an image related mapping is supported by FileObject-> SectionObjectPointer->ImageSectionObject. DataSectionObject and ImageSectionObject are a CONTROL_AREA structure that contains a list of subsections which manage resident pages list. So there are two set of pages - one for a data map and another for an image map.

While on many UNIX systems pages are shared between image and data mappings, e.g. for macOS data and image mappings share the same data structure vnode->v_un.vu_ubcinfo->ui_control->moc_object->memq where memq is a list of resident pages.

When you modify a mapped executable image a private COW copy of a page is created and it is backed by a page file. It doesn’t make sense to modify overlapped data for a mapped image file on a page beyond valid data length for a section terminating on this page. If you want modify an image file section do this starting from a valid mapping address for a section as defined in a PE header. You don’t have an aliasing problem in this case as you have in virtually addressed CPU caches as one copy of the data is not used during file execution, this is garbage from the system point of view.

To my knowledge there is no data access occurring here. I am logging IRPs when an exe is loaded for execution. My understanding is that Windows maps the exe into cache for the duration of its execution and uses the exe itself to back that memory rather than using the paging file. It has to be that regardless of the alignment, that these pages cover the entire exe once, without overlaps.

Even if it were data access loading it into a different cache, why would Windows using paging io to load all but 2 4K pages and one 16K contiguous block, and then the data access would load the 3 chunks that were left, but 3K out of alignment with the rest of the file? Does Windows know that certain parts of the image are not needed, because they are data components, so it leaves them, but as execution begins those are loaded into the data cache? Why would the data cache not then be page aligned like it is with normal data files? And why is this not consistent behavior exhibited by all exe’s? Many of my test cases loaded entirely in page aligned chunks, albeit not sequential from the start of the exe.

_Ron

Before you continue increasing the post’s entropy you should apprehend the following

  • Windows doesn’t map executable files in the data cache for code execution. An executable file is mapped to a user space portion of a process address space.
  • Executable files are mapped not for the sake of speed up (i.e. caching) but to allow a CPU to fetch instructions and data. A CPU can’t read data from a storage. Speed up is a side effect of the mapping.
  • Executable files are not cached like data files. The illusion of caching is due to the delayed page reclaiming.
  • An executable file consist of the sections.
  • Each section in an executable file is aligned on a boundary which is a multiple of a constant defined in a PE header. This constant is 0x200 for most of the cases, i.e. a disk sector size.
  • When being mapped in a process address space for execution each section is aligned on a boundary which is a multiple of another constant defined in a PE header. This constant should be a multiple of a page size and is usually defined as 0x1000, i.e. a page size.
  • If a process virtual address range backed by an executable file mapping is being modified a Copy-On-Write mechanism is triggered making a process private copy for modified page(s). These pages are backed by a paging file.
  • Windows has a prefetch mechanism that brings pages in memory before they are being accessed.
  • Windows collects information on executable file access patterns and uses it for prefetching.
  • When an executable file is loaded some external references and relocations must be resolved. This initiates memory I/O.

Having read the above, you should know

  • for an executable image an offset for paging IO is a multiple of a disc sector
  • executable file mappings don’t use data cache
  • Windows knows file access patterns and uses them for prefetching

I appreciate the information. If you are aware of formal documentation on this (e.g. an MS web link or a section in Windows Internal), I would love to read more on this process.

Thanks for your help

_Ron

The only documentation (beside the Windows kernel source) I can recommend is
Enrico Martignetti’s “What makes it Page?: The Windows 7 Virtual Memory Manager”. Chapters 38.4.2 - 38.4.3 (pp 363-369) are relevant to this discussion.

Thanks, I will make a note to look for a copy tonight