Understanding Cache

NTFSD
1

Hello,
Can anyone explain to me the meaning of cache :slight_smile:

If I am copying a file from one disk to another with buffering (using cache), and the file is like 20GB.

  1. Is the whole file going to be cached?
  2. If yes where is it going to be cached? (500MB-Local memory, 40GB-harddisk)
  3. If no, who is stopping it from being cached?
  4. Is there a way to partially cache a file?
  5. Does explorer ‘knows’ that when there is low system memory then the copy is going to be non-buffered?

Igal

2

Hello guys,

I have found this link on MS sight explaining the problems that I have experienced.

http://blogs.technet.com/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

Igal

3

> I have found this link on MS sight explaining the problems that I have experienced.

http://blogs.technet.com/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

This link seems to overlook the most important thing, namely, FILE_FLAG_NO_BUFFERING flag. Although it does say that Cache Manager’s involvement results in overhead on large reads/writes, it does not seem to offer any meaningfull solution. It says that one should use CreateFile(), ReadFile() and WriteFile() calls, instead of CopyFile(), but it does not say that one should specify FILE_FLAG_NO_BUFFERING in CreateFile() call, i.e. the only thing that really matters in this context, because this flag disables cacheing for a given file…

Anton Bassov

4

Anton,

thank you for your reply.

Do you have any Idea how explorer activates the copy operation(copy,paste) w/wo the flag?

Actually, I have experienced this problem during copy of a large file from my file system, but when copied large file from a USB disk with NTFS to local disk with NTFS, no drop in system memory.

Does explorer has the ability to identify 2 NTFSs and then activate the copy with FILE_FLAG_NO_BUFFERING?

I will really appreciate your answer,

Igal

5

You might find FILEMON from Microsoft/sysinternals helpful. It monitors
file based I/O on a machine. In this particular case, it will not be
the easiest thing to track, as “copy” is not single IRP, but, in
general, it could help answer a lot of your questions.

Just google FILEMON, and it almost certainly will be the first link.

Good luck,

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
igalk013@013.net.il
Sent: Friday, August 24, 2007 14:23
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Understanding Cache

Anton,

thank you for your reply.

Do you have any Idea how explorer activates the copy
operation(copy,paste) w/wo the flag?

Actually, I have experienced this problem during copy of a large file
from my file system, but when copied large file from a USB disk with
NTFS to local disk with NTFS, no drop in system memory.

Does explorer has the ability to identify 2 NTFSs and then activate the
copy with FILE_FLAG_NO_BUFFERING?

I will really appreciate your answer,

Igal

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@evitechnology.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

6

Wrong utility. It is Process Monitor. It combines the old filemon and
regmon into a single program with more features.

–
David J. Craig
Engineer, Sr. Staff Software Systems
Broadcom Corporation

“Martin O’Brien” wrote in message
news:xxxxx@ntfsd…
You might find FILEMON from Microsoft/sysinternals helpful. It monitors
file based I/O on a machine. In this particular case, it will not be
the easiest thing to track, as “copy” is not single IRP, but, in
general, it could help answer a lot of your questions.

Just google FILEMON, and it almost certainly will be the first link.

Good luck,

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
igalk013@013.net.il
Sent: Friday, August 24, 2007 14:23
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Understanding Cache

Anton,

thank you for your reply.

Do you have any Idea how explorer activates the copy
operation(copy,paste) w/wo the flag?

Actually, I have experienced this problem during copy of a large file
from my file system, but when copied large file from a USB disk with
NTFS to local disk with NTFS, no drop in system memory.

Does explorer has the ability to identify 2 NTFSs and then activate the
copy with FILE_FLAG_NO_BUFFERING?

I will really appreciate your answer,

Igal

—
NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@evitechnology.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

7

Better idea.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of David J. Craig
Sent: Friday, August 24, 2007 17:17
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Understanding Cache

Wrong utility. It is Process Monitor. It combines the old filemon and
regmon into a single program with more features.

–
David J. Craig
Engineer, Sr. Staff Software Systems
Broadcom Corporation

“Martin O’Brien” wrote in message
news:xxxxx@ntfsd…
You might find FILEMON from Microsoft/sysinternals helpful. It monitors
file based I/O on a machine. In this particular case, it will not be
the easiest thing to track, as “copy” is not single IRP, but, in
general, it could help answer a lot of your questions.

Just google FILEMON, and it almost certainly will be the first link.

Good luck,

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
igalk013@013.net.il
Sent: Friday, August 24, 2007 14:23
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Understanding Cache

Anton,

thank you for your reply.

Do you have any Idea how explorer activates the copy
operation(copy,paste) w/wo the flag?

Actually, I have experienced this problem during copy of a large file
from my file system, but when copied large file from a USB disk with
NTFS to local disk with NTFS, no drop in system memory.

Does explorer has the ability to identify 2 NTFSs and then activate the
copy with FILE_FLAG_NO_BUFFERING?

I will really appreciate your answer,

Igal

—
NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@evitechnology.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

—
NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@evitechnology.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

8

Well MM and David,

First thank you for the advice, I have implemented it, and saw that on both operations:

  1. copy 4 GB file from USB-NTFS disk to local-NTFS disk
  2. copy 4 GB file from USB-myFS disk to local-NTFS disk

There are a lot of read,write, fastio (I have expected it, as I saw them during the FS development)

So I tried to filter CreateFile. And I have noticed that there are a lot of create file operations on the source file and then one create file operation on the destination file. I have also filtered the close file operation to verify that there are no parallel access to the file. from the “Detail” of the create file operation I saw that the only difference between both copy operations, is that in the case of an USB-NTFS disk after openeing the source file for read, the file is opened again to set the “Desired Access” with: Read Attributes, Synchronize ( I saw nothing that deals with sync in CreateFile)

Does it mean anything or am I looking under the wrong flashlight?

Igal