Assessing the cache status of a file

NTDEV
1

Hi,

I wonder if you guys can help me with the following. This may sound like a user-land problem, but bear with me for a bit.

Given a file name, I’d like to be able to tell if the file (or its head part) is currently present in Windows cache. This in the context of cache-friendly massive copying application (think - a backup proggy), and the idea is to utilize the cache if a file is in it, and fall back to the unbuffered IO otherwise. In other words - try and not put anything into cache, but still read from it if possible.

There is no Cache Manager API or at least nothing public - this much I know. So this leaves me with two options -
(a) to write some kernel component to poke around the Cache and report the results to the user-space process
(b) to try and understand cache status of a file by timing the execution of Win API calls

–

The (a) I haven’t touched yet, but I did run some tests along the (b) lines.

The most obvious option is to open the file and read a small amount of data from it. Do it twice, once - with FLAG_FILE_NO_BUFFERING, and second time - without. Compare the read times and if latter is substantially smaller than former, then the read block was in the cache. If the file appears to be cached, then keep reading it in small-ish chunks until the read time jumps indicating the cache miss. Re-open file unbuffered and continue reading.

This works, but it still pushes small amount of data into the cache, and it becomes a problem when reading large amounts of small files. The cache gets thrashed.

–

Now, if I haven’t lost you yet, the question. Can anyone think of an File I/O API call that has an IF (hinging on the cache presence) in it, and that does NOT change the cache?

It would take me some time to find my copy of SoftICE and get all set for stepping through the kernel… which is something that I am fully intended to do. But meanwhile if anyone has any thought or alternative suggestions, I am all ears.

Cheers,
Alex

2

> (a) to write some kernel component to poke around the Cache and report the

results to the user-space process

The problem that you have here is that that the cache state is transient.
Even if you had a way to ask, “is this chunk in the cache” by the time you
accessed it it might have been paged out from the cache.

Also, trying to poke through the cache will soon turn into a major
nightmare. The structures are all undocumented and change from release to
release, so you’d end up with a mess that sort of works some of the time.

Now, if I haven’t lost you yet, the question. Can anyone think of an File
I/O API call that has an IF (hinging on the cache >presence) in it, and
that does NOT change the cache?

You might find a kernel mode API CcIsFileCached that will get you excited,
but that only makes sense when called from the file system driver itself
(and it only tells you that caching has been initialized for the file, not
what regions are currently in the cache).

It would take me some time to find my copy of SoftICE and get all set for
stepping through the kernel… which is something >that I am fully intended
to do.

Don’t bother, use two machines and WinDBG instead and save yourself lots of
pain.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntdev…
> Hi,
>
> I wonder if you guys can help me with the following. This may sound like a
> user-land problem, but bear with me for a bit.
>
> Given a file name, I’d like to be able to tell if the file (or its head
> part) is currently present in Windows cache. This in the context of
> cache-friendly massive copying application (think - a backup proggy), and
> the idea is to utilize the cache if a file is in it, and fall back to the
> unbuffered IO otherwise. In other words - try and not put anything into
> cache, but still read from it if possible.
>
> There is no Cache Manager API or at least nothing public - this much I
> know. So this leaves me with two options -
> (a) to write some kernel component to poke around the Cache and report the
> results to the user-space process
> (b) to try and understand cache status of a file by timing the execution
> of Win API calls
>
> –
>
> The (a) I haven’t touched yet, but I did run some tests along the (b)
> lines.
>
> The most obvious option is to open the file and read a small amount of
> data from it. Do it twice, once - with FLAG_FILE_NO_BUFFERING, and second
> time - without. Compare the read times and if latter is substantially
> smaller than former, then the read block was in the cache. If the file
> appears to be cached, then keep reading it in small-ish chunks until the
> read time jumps indicating the cache miss. Re-open file unbuffered and
> continue reading.
>
> This works, but it still pushes small amount of data into the cache, and
> it becomes a problem when reading large amounts of small files. The cache
> gets thrashed.
>
> –
>
> Now, if I haven’t lost you yet, the question. Can anyone think of an File
> I/O API call that has an IF (hinging on the cache presence) in it, and
> that does NOT change the cache?
>
> It would take me some time to find my copy of SoftICE and get all set for
> stepping through the kernel… which is something that I am fully intended
> to do. But meanwhile if anyone has any thought or alternative suggestions,
> I am all ears.
>
> Cheers,
> Alex
>
>

3

Scott, thanks for the reply. Much appreciated.

Even if you had a way to ask, “is this chunk in the cache” by the time you accessed it it might have been paged out from the cache.

I understand this of course. The goal is to make the best effort avoiding any cache writes.

The structures are all undocumented and change from release to release, so you’d end up with a mess that sort of works some of the time.

I suspected as much.

… CcIsFileCached … and it only tells you that caching has been initialized for the file, not what regions are currently in the cache

This is actually what I am after at the moment. I just want to know for sure if the file is NOT in the cache. A more elaborate and proper implementation is do a per-chunk tracking, but that’s substantially more effort and it will need to wait.

In any case, this is a solid lead, thanks. Can you think of any public user-space API call that makes use of it? Essentially, I’d like to do something like

t0 = timestamp();

if (timestamp() - t0 > threshold)
// likely not “in a cache”, i.e. CcIsFileCached() is false
else
// likely cached, i.e. CcIsFileCached() is true

This will be a subject to false positives and negatives, the ‘threshold’ will need to be calibrated per-machine and per-load, but I’m just trying to stay in userland and get by on as little coding as possible :slight_smile:

4

Have you tried opening the files with the sequential access flag? The cache manager will then perform unmap-behind as you are processing each file (assuming you actually do this sequentially), so you won’t be polluting the cache.

If you still see system cache grow very large with the sequential access flag, it might be due to NTFS metadata (e.g. the MFT). You can check whether this is the case using RamMap.exe from sysinternals. If it turns out to be due to the MFT there might be a few more things to try, but it would be nice to understand the exact reason first.

Thanks,
Pavel

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@poneyhot.org
Sent: Thursday, November 04, 2010 8:44 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Assessing the cache status of a file

Hi,

I wonder if you guys can help me with the following. This may sound like a user-land problem, but bear with me for a bit.

Given a file name, I’d like to be able to tell if the file (or its head part) is currently present in Windows cache. This in the context of cache-friendly massive copying application (think - a backup proggy), and the idea is to utilize the cache if a file is in it, and fall back to the unbuffered IO otherwise. In other words - try and not put anything into cache, but still read from it if possible.

There is no Cache Manager API or at least nothing public - this much I know. So this leaves me with two options -
(a) to write some kernel component to poke around the Cache and report the results to the user-space process
(b) to try and understand cache status of a file by timing the execution of Win API calls

–

The (a) I haven’t touched yet, but I did run some tests along the (b) lines.

The most obvious option is to open the file and read a small amount of data from it. Do it twice, once - with FLAG_FILE_NO_BUFFERING, and second time - without. Compare the read times and if latter is substantially smaller than former, then the read block was in the cache. If the file appears to be cached, then keep reading it in small-ish chunks until the read time jumps indicating the cache miss. Re-open file unbuffered and continue reading.

This works, but it still pushes small amount of data into the cache, and it becomes a problem when reading large amounts of small files. The cache gets thrashed.

–

Now, if I haven’t lost you yet, the question. Can anyone think of an File I/O API call that has an IF (hinging on the cache presence) in it, and that does NOT change the cache?

It would take me some time to find my copy of SoftICE and get all set for stepping through the kernel… which is something that I am fully intended to do. But meanwhile if anyone has any thought or alternative suggestions, I am all ears.

Cheers,
Alex

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

5

>In any case, this is a solid lead, thanks. Can you think of any public

user-space API call that makes use of it? Essentially, I’d >like to do
something like

Sorry, don’t know of any. Usually it’s a, “don’t care” type of situation.

My only lead that I could give you at this point would be to look into the
Superfetch support that’s been added to the O/S on Vista and later. That
includes new file info classes from the Mm and a file system filter driver
(fileinfo) that report information about various memory usage things. While
this is all entirely undocumented and I can’t actually tell you if any of
the info available from these is useful (I don’t know, I’ve never actually
looked in detail) that’s probably your best bet.

I’ll repeat again that it’s all undocumented and subject to change, but it’s
all I can think of.

Good luck!

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntdev…
> Scott, thanks for the reply. Much appreciated.
>
>> Even if you had a way to ask, “is this chunk in the cache” by the time
>> you accessed it it might have been paged out from the cache.
>
> I understand this of course. The goal is to make the best effort avoiding
> any cache writes.
>
>> The structures are all undocumented and change from release to release,
>> so you’d end up with a mess that sort of works some of the time.
>
> I suspected as much.
>
>> … CcIsFileCached … and it only tells you that caching has been
>> initialized for the file, not what regions are currently in the cache
>
> This is actually what I am after at the moment. I just want to know for
> sure if the file is NOT in the cache. A more elaborate and proper
> implementation is do a per-chunk tracking, but that’s substantially more
> effort and it will need to wait.
>
> In any case, this is a solid lead, thanks. Can you think of any public
> user-space API call that makes use of it? Essentially, I’d like to do
> something like
>
> t0 = timestamp();
>
> if (timestamp() - t0 > threshold)
> // likely not “in a cache”, i.e. CcIsFileCached() is false
> else
> // likely cached, i.e. CcIsFileCached() is true
>
> This will be a subject to false positives and negatives, the ‘threshold’
> will need to be calibrated per-machine and per-load, but I’m just trying
> to stay in userland and get by on as little coding as possible :slight_smile:
>

6

If the file is in cache, I believe non-buffered requests will still fetch it from the cache. Non-buffered writes should be updating any pages currently present in the cache. This is because the cache needs to be coherent no matter how the files are accessed.

So open the file in non-buffered mode and forget about trying to over-optimize it.

7

Also, if you’ll even be able to pull 4 GB out of cache, that will only save about 1 minute of reading with modern disks. Even not too modern ones.

8

wrote in message news:xxxxx@ntdev…
> If the file is in cache, I believe non-buffered requests will still fetch
> it from the cache.

This isn’t the way that Windows file systems handle this case. Instead, what
they do is flush the cache to disk before performing the non-cached
transfer. See the FASTFAT source:

//
// If this is a noncached transfer and is not a paging I/O, and
// the file has been opened cached, then we will do a flush
here
// to avoid stale data problems. Note that we must flush
before
// acquiring the Fcb shared since the write may try to acquire
// it exclusive.
//
// The Purge following the flush will garentee cache coherency.
//

if (NonCachedIo && !PagingIo &&
(FileObject->SectionObjectPointer->DataSectionObject !=
NULL)) {

//
// We need the Fcb exclsuive to do the CcPurgeCache
//

if (!FatAcquireExclusiveFcb( IrpContext, FcbOrDcb )) {

DebugTrace( 0, Dbg, “Cannot acquire FcbOrDcb = %08lx
shared without waiting\n”, FcbOrDcb );

try_return( PostIrp = TRUE );
}

FcbOrDcbAcquired = TRUE;
FcbAcquiredExclusive = TRUE;

//
// Preacquire pagingio for the flush.
//

ExAcquireSharedStarveExclusive(
FcbOrDcb->Header.PagingIoResource, TRUE );

CcFlushCache( FileObject->SectionObjectPointer,
WriteToEof ? &FcbOrDcb->Header.FileSize :
&StartingByte,
ByteCount,
&Irp->IoStatus );

ExReleaseResourceLite( FcbOrDcb->Header.PagingIoResource );

That’s from the write path, but the same logic is also in the read path.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntdev…
> If the file is in cache, I believe non-buffered requests will still fetch
> it from the cache. Non-buffered writes should be updating any pages
> currently present in the cache. This is because the cache needs to be
> coherent no matter how the files are accessed.
>
> So open the file in non-buffered mode and forget about trying to
> over-optimize it.
>

9

> the idea is to utilize the cache if a file is in it, and fall back to the unbuffered IO otherwise.

In other words - try and not put anything into cache, but still read from it if possible.

If you want to speed up frequent reads and writes to the same parts of the file you should do buffered IO in order to avoid slowdown resulting from disk access. If you want to speed up reads and writes that involve large amounts of data you should do unbuffered IO in order to avoid slowdown resulting from synchronous nature of MM’s operations.

What you are speaking about is more than likely to get you the worst possible combination of buffered and unbuffered IO…

Anton Bassov

10

Pavel, thanks for the reply.

Have you tried opening the files with the sequential access flag? The cache manager will then perform unmap-behind as you are processing each file (assuming you actually do this sequentially), so you won’t be polluting the cache.

I knew about the flag, but didn’t really think through its semantics. Do you know of any pages explaining the interaction of SCAN_SEQUENTIAL and overlapped I/O?

If not, then just a concrete question -

do I understand correctly that if I am processing file A with SCAN_SEQUENTIAL, stop in its middle, close the handle and start processing file B the same way, then file C, D, E, etc., then my induced cache usage will still be capped? With the cap being my total buffer size (across all pending overlapped ReadFile calls) plus some read-ahead overhead.

Also, do you know if there is any principal difference in treatment of SCAN_SEQUENTIAL between XP and later Windows versions? … as RamMap.exe is Vista+ only.

11

@Scott - thanks, will have a look at Superfetch.

@Alex and @Anton -

Based on cursory testing ReadFile performs along these lines:
* Without NO_BUFFERING and with cached content ~ 50 time units
* Without NO_BUFFERING and a cache miss ~ 3000 time units
* With NO_BUFFERING ~ 2500 time units

So basically the idea is that some time can be saved by reading from the cache until it is exhausted and then switching to unbuffered IO to not spend any time on populating the cache.

12

> Based on cursory testing ReadFile performs along these lines:

* Without NO_BUFFERING and with cached content ~ 50 time units
* Without NO_BUFFERING and a cache miss ~ 3000 time units
* With NO_BUFFERING ~ 2500 time units

This also depends on IO operation size a lot.

So basically the idea is that some time can be saved by reading from the cache

Backup apps deal with huge amounts of data. For this, ~4GB of cache on modern machines is nothing.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

13

> This also depends on IO operation size a lot.

True, of course. However based on some testing buffer sizes from 512K to 8M deliver about the same overall performance for HDD, SSD and USB drives. Outside of this range performance starts to drop for some drive types, but for some it stays about the same.

Backup apps deal with huge amounts of data.

That would be traditional (scheduled) backups that copy everything. There are also real-time backups that effectively do incremental updates of a smaller subset of changing files. For these optimizing the IO performance is quite important.

14

> So basically the idea is that some time can be saved by reading from the cache until it is exhausted

and then switching to unbuffered IO to not spend any time on populating the cache.

Yes, but what about writes??? Look - every time you do unbuffered write parts of your cache will get invalidated and subsequently reloaded into memory. This is why I am saying that, depending on IO pattern, you may get the worst of both IO types if you mix them up…

Anton Bassov

15

OK, if doing this hack will save you 1, or 10 minutes for your incremental backup, is it really worth writing a complicated kludge?

16

At the risk of stating the obvious (though nobody else has): This topic belongs on NTFSD, not NTDEV.

Peter
OSR

17

Peter, my apologies. I put it here because it’s more of a Cache Manager question rather than plain FS. Please move the thread if necessary.

Alex - if this saves me 5 seconds I will be ecstatic :slight_smile: See below.

Anton - All this is in the context incremental real-time *delta* backups. You touch the file, and the *change* is pushed to the backup copy within seconds after. For example, opening a page in Firefox induces a small change in places.sqlite file. The file itself is in tens of megabytes, so this is how much is read, but the write is limited to only few kilobytes. Makes sense?

18

> The file itself is in tens of megabytes, so this is how much is read, but the write is limited to only

few kilobytes. Makes sense?

Actually, the only thing that seems to make sense to me under these circumstances is FILE_FLAG_NO_BUFFERING. I really don’t see any reason why you want to introduce buffering here (unless you are just desperate to slow things down on purpose, of course)…

if this saves me 5 seconds I will be ecstatic

Bypassing CM/MM (i.e. something that FILE_FLAG_NO_BUFFERING implies) on large data reads/writes may save you more than that…

Anton Bassov

19

> Actually, the only thing that seems to make sense to me under these

circumstances is FILE_FLAG_NO_BUFFERING. I really don’t see any
reason why you want to introduce buffering here (unless you are just
desperate to slow things down on purpose, of course)…

Well, simply because if the (part of the) file is in the cache, it can be scanned for changes much faster, so no reason to utilize this. Please have another look at my original post. I want to read what’s in cache, and then when it’s exhausted, switch to the unbuffered read. Not sure how else I could explain it.

20

> … so no reason to utilize this …

It should be “… no reason NOT to utilize …”.