Hello experts,
Is there any way for the IO manager to bypass the filesystem to serve data
to the applications? In other words, can IO manager directly fetch data from
cache manager, without even using FastIO?
Bedanto
The I/O Manager cannot.
However the Memory Manager can (and does). That’s called “memory
mapping” and is used by Notepad.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Bedanto
Sent: Wednesday, February 22, 2006 1:30 AM
To: ntfsd redirect
Subject: [ntfsd] cache manager
Hello experts,
Is there any way for the IO manager to bypass the filesystem to serve
data to the applications? In other words, can IO manager directly fetch
data from cache manager, without even using FastIO?
Bedanto
— Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17 You are currently subscribed
to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a
blank email to xxxxx@lists.osr.com
No, IO Manager cannot.
>However the Memory Manager can (and does). That’s called “memory mapping”
and is used by Notepad.
Okay, this is good, so when a memory mapped file is created, all IO is
directed towards the memory image of the same. But it has to write it back
to the disk at some time.
I guess a memory mapped file is like a normal file, where many appplications
can read and write to it, if the ACLs are proper, and the FS won’t even have
a clue of these operations.
Right. In order to create a memory mapped file you have to first open
the file (which does an access check) and then take the file handle and
pass it to the file mapping API; that gives you back a section handle
that can then be used to map a region (“view”) of the file.
Changes to the data in the memory mapped view are immediately in the
cache (this is really a common piece of memory between the file system
cache and the memory mapped file) so that a read from that file will
show the update. Eventually those changes will be written back to the
underlying file system using paging I/O operations.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Bedanto
Sent: Wednesday, February 22, 2006 1:54 AM
To: ntfsd redirect
Subject: Re: [ntfsd] cache manager
However the Memory Manager can (and does). That’s called “memory
mapping” and is used by Notepad.
Okay, this is good, so when a memory mapped file is created, all IO is
directed towards the memory image of the same. But it has to write it
back to the disk at some time.
I guess a memory mapped file is like a normal file, where many
appplications can read and write to it, if the ACLs are proper, and the
FS won’t even have a clue of these operations.
— Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17 You are currently subscribed
to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a
blank email to xxxxx@lists.osr.com
> However the Memory Manager can (and does). That’s called “memory mapping”
and is used by Notepad.
The question was if I/O manager can bypass the file system. The answer is “no”.
memory mapping bypasses cached requests only.
Nothing bypasses file system to get the file data.
L.
I am not sure whether I got this paging IO thing correct or not.
Say I open a file in notepad, which memory maps it. I do changes to the same
file, add something, delete some stuff etc, all these operations are done in
memory.
So when I press save, how is the call routed down, through IO manager or
through memory manager?
Sorry if it is a stupid question, but my knowledge is poor and I am here to
learn.
…also
ladislav, what did you mean by “memory mapping bypasses cached requests
only.”
I agree that the answer to his literal question was “no”. However, I
was trying to read further into his question (and to me it seemed he was
asking "is there a path into the data that is not read and write.)
Since this is an area of constant confusion, I was trying to avoid such.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ladislav Zezula
Sent: Wednesday, February 22, 2006 2:50 AM
To: ntfsd redirect
Subject: Re: [ntfsd] cache manager
However the Memory Manager can (and does). That’s called “memory
mapping”
and is used by Notepad.
The question was if I/O manager can bypass the file system. The answer
is “no”.
memory mapping bypasses cached requests only.
Nothing bypasses file system to get the file data.
L.
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
Save As is not a single operation - it is a large SET of operations.
You’d be best served if you went and took FileSpy and watched the set of
operations. Set up a file on a separate drive - monitor only that drive
- open it with notepad, modify it and then save it.
Notepad is far trickier than you think it is. You might even notice
that the file is not open while you are editing it (it isn’t and there
are good historical reasons why that is the case.) The file will only
be opened once you try to save it.
Computer systems seldom work the way you think they do, which is why
observing the behavior can often be an “eye opening” experience. Try it
- you’ll be surprised.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Bedanto
Sent: Wednesday, February 22, 2006 3:21 AM
To: ntfsd redirect
Subject: Re: [ntfsd] cache manager
I am not sure whether I got this paging IO thing correct or not.
Say I open a file in notepad, which memory maps it. I do changes to the
same file, add something, delete some stuff etc, all these operations
are done in memory.
So when I press save, how is the call routed down, through IO manager or
through memory manager?
Sorry if it is a stupid question, but my knowledge is poor and I am here
to learn.
— Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17 You are currently subscribed
to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a
blank email to xxxxx@lists.osr.com
> ladislav, what did you mean by "memory mapping bypasses
cached requests only."
Well maybe not quite well explained, I mean:
- “normal” file approach (CreateFile/ReadFile/WriteFile/CloseHandle)
You’ll see both cached and non-cached read/writes in FS or FSD.
Note that this depends on file system, I can imagine a 3rd party FS
which does not use cache manager, do this is not a dogma.
- “mapped file” approach
(CreateFile/CreateFileMapping/MapViewOfFile/UnmapViewOfFile)
Memory mapping sends down paging I/O requests,
they are noncached. It’s probable that you will not see
a cached read/write at all.
Note that this depends on what the application exactly does,
so it’s not a dogma as well
L.
notepad is quite a surprise when u debug it. it memory maps files
afteropening them and closes the original one, thus if u try to find open
handles, you wont find one to any file on the disk.
when u save a file, then it opens the same file and saves the data and
closes it.
it would be difficult to track the path of a particu;ar save operation in
any application, as in all probability it is a SEQUENCE of operations and
not an atomic one.
what are you trying to achieve, by the way???
>what are you trying to achieve, by the way???
I am trying to rewrite filemon myself, just for learning purposes. After
reading Nagar’s book, I got confused whether all IO will route through me of
not. Seems that it will not. 
BTW, in filemon/filespy I see operations where the filename is C:DASD, what
is that?
Tony, I have already tried out what you said, and had posted the initial
mail afte that.
ladislav,
- “mapped file” approach
(CreateFile/CreateFileMapping/MapViewOfFile/UnmapViewOfFile)
Memory mapping sends down paging I/O requests,
they are noncached. It’s probable that you will not see
a cached read/write at all.
Note that this depends on what the application exactly does,
so it’s not a dogma as well
waht can u explain this point furhter please…
> waht can u explain this point furhter please…
This topic is well explained in the Rajeev Nagar’s book.
(you wrote you’ve read it)
I am not unwilling to explain it, I am just too lazy to
rewrite the particular part of the book into mail 
Note that this depends on what the application exactly does,
so it’s not a dogma as well
An application may either
-
only use cached reads for all access to
the file data (only ReadFile/WriteFile comes into play) -
only use memory mapped files for accessing file data
-
may combine both approaches
I hope I have explained it well, if not, tell me what in particular
is not clear to you.
L.
>This topic is well explained in the Rajeev Nagar’s book.
(you wrote you’ve read it)
I am not unwilling to explain it, I am just too lazy to
rewrite the particular part of the book into mail
Can you tell me where to look for it. there are three who chapters on cache
manager, and one on VMM, and there is way too much information to grasp in
one go, that too for a newbie in drivers.
An application may either
- only use cached reads for all access to
the file data (only ReadFile/WriteFile comes into play)
In this case, would a file system filter driver be able to catch these calls
through FastIo routines?
- only use memory mapped files for accessing file data
Now, this I am sure has nothing to do with file IO, so definitely they won’t
come down to the FSFD, am I correct? So if I open a file and make a memory
mapped view of this, and any other application can read off this file, and
the FSFD won’t have a clue of this, this is more sort of a shared memory
technique to me than file IO!!!
I tool, like filemon needs to log file operatins by aplications, and I am
more concerned, whether my driver can be bypassed in any way to read off
cached data by the IO manager. Memory mapped files, I don’t think is in my
current scope, and I don’t think it is even possible to get these calls,
unless we hook some routines. Am I correct in this?
I hope I have explained it well, if not, tell me what in particular
is not clear to you.
The above points, I mentioned, needs to be authenticated.
Thanks for all the help extended, I rally appreaciate this community.
If you want to catch all I/O, you should look at non-cached I/O - this
will include application non-cached I/O (which is used by databases, for
example) and paging I/O (which will include data written from VM -
including the file system data cache as well as application level memory
mapped files).
Fast I/O is an optional FSD interface and may not even be used for all
I/O operations (for example, it is common to wait until the first I/O to
set up caching which is required for using fast I/O read or write.)
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Bedanto
Sent: Wednesday, February 22, 2006 5:50 AM
To: ntfsd redirect
Subject: Re: [ntfsd] cache manager
This topic is well explained in the Rajeev Nagar’s book.
(you wrote you’ve read it)
I am not unwilling to explain it, I am just too lazy to
rewrite the particular part of the book into mail
Can you tell me where to look for it. there are three who chapters on
cache manager, and one on VMM, and there is way too much information to
grasp in one go, that too for a newbie in drivers.
An application may either
- only use cached reads for all access to
the file data (only ReadFile/WriteFile comes into play)
In this case, would a file system filter driver be able to catch these
calls through FastIo routines?
- only use memory mapped files for accessing file data
Now, this I am sure has nothing to do with file IO, so definitely they
won’t come down to the FSFD, am I correct? So if I open a file and make
a memory mapped view of this, and any other application can read off
this file, and the FSFD won’t have a clue of this, this is more sort of
a shared memory technique to me than file IO!!!
I tool, like filemon needs to log file operatins by aplications, and I
am more concerned, whether my driver can be bypassed in any way to read
off cached data by the IO manager. Memory mapped files, I don’t think is
in my current scope, and I don’t think it is even possible to get these
calls, unless we hook some routines. Am I correct in this?
I hope I have explained it well, if not, tell me what in particular
is not clear to you.
The above points, I mentioned, needs to be authenticated.
Thanks for all the help extended, I rally appreaciate this community.
— Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17 You are currently subscribed
to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a
blank email to xxxxx@lists.osr.com
> Can you tell me where to look for it. there are three who chapters on cache manager,
and one on VMM, and there is way too much information to grasp in one go, that too
for a newbie in drivers.
You must learn this informations anyway, it is essential
for every filter/FS driver developer.
If you are not able to search the information in the book,
you are definitely not able to write drivers at all.
I respect you are newbie, but writing FS drivers or filters is not like
writing a “hello, world” program. You have to learn the principles,
and concepts of it.
In this case, would a file system filter driver be able to catch these
calls through FastIo routines?
Yes. But if there’s filter attached to a volume who returns FALSE from
FastIo calls, the driver receives only IRP based read/write requests.
>2) only use memory mapped files for accessing file data
Now, this I am sure has nothing to do with file IO, so definitely
they won’t come down to the FSFD, am I correct?
No. The only way how to get file data is to read them through the file system.
Otherwise you can only read raw disk sectors and this is another chapter.
Again, read the Nagar’s book to learn how it works.
Memory mapped files, I don’t think is in my current scope, and I don’t
think it is even possible to get these calls, unless we hook some routines.
Am I correct in this?
No. You will see paging I/O reads or writes in case that the data are not already
in the memory manager’s pool of mapped files.
L.
>Fast I/O is an optional FSD interface and may not even be used for all I/O
operations (for example, it is common to wait until the first I/O to >set up
caching which is required for using fast I/O read or write.)
Tony, I am not sure whether I understand what you say. can you elaborate the
portion in the braces a bit more please, else give me some pointers, and I
think i can find out for myself.
You must learn this informations anyway, it is essential
for every filter/FS driver developer.
If you are not able to search the information in the book,
you are definitely not able to write drivers at all.
I am trying my best, but it is not easy to grasp these things, till you take
a few OSR classes or spend a few years looking at code. Unfortunately, the
option of attending these seminars are NULL for me.
I respect you are newbie, but writing FS drivers or filters is not like
writing a “hello, world” program. You have to learn the principles,
and concepts of it.
That is why, I expect help from you people. I have studied Filemon code, and
filespy code, and sfilter code, and hence decided to rewrite the filemon
tool myself from scratch, so that I could have a better grasp of things.
Yes. But if there’s filter attached to a volume who returns FALSE from
FastIo calls, the driver receives only IRP based read/write requests.
Well, that might happen for third party drivers, if I know correctly, all
microsoft file system drivers, do support FastIo. Please correct me if I am
mistaken. More over, if a third party driver doesn’t support FastIo, their
performance would be so much degraded, that they better use it instead, am I
correct in this aspect?
Memory mapped files, I don’t think is in my current scope, and I don’t
think it is even possible to get these calls, unless we hook some
routines.
Am I correct in this?
No. You will see paging I/O reads or writes in case that the data are not
already
in the memory manager’s pool of mapped files.
But these paging Io calls would be from a different process context, not the
original application, correct? If this is true, there will be no way to
guess who the actual benifactor is. Or is there a way to figure that out
too?
Actually some guru once told me, that Filemon cannot see all file based io.
A much bette approach would be to hook Nt* calls in kernel mode through SSDT
hooking. This, he justified, would also take care of those calls that are
directed to the cache manager eventually. Is he correct in this?
Please give your advices.
Once again, I thank you all for the help you extend.
L.
Normally, caching is not established until first I/O because the call to
set up (CcInitializeCacheMap) is “expensive” when many file open
operations do not involve I/O.
Fast I/O is a misnomer - it is not necessarily any faster than normal
I/O these days and I have seen it be slower to support Fast I/O. Do not
read too much into a 10+ year old name.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Bedanto
Sent: Wednesday, February 22, 2006 7:32 AM
To: ntfsd redirect
Subject: Re: [ntfsd] cache manager
Fast I/O is an optional FSD interface and may not even be used for all
I/O operations (for example, it is common to wait until the first I/O to
set up caching which is required for using fast I/O read or write.)
Tony, I am not sure whether I understand what you say. can you elaborate
the portion in the braces a bit more please, else give me some pointers,
and I think i can find out for myself.
You must learn this informations anyway, it is essential
for every filter/FS driver developer.
If you are not able to search the information in the book,
you are definitely not able to write drivers at all.
I am trying my best, but it is not easy to grasp these things, till you
take a few OSR classes or spend a few years looking at code.
Unfortunately, the option of attending these seminars are NULL for me.
I respect you are newbie, but writing FS drivers or filters is not like
writing a “hello, world” program. You have to learn the principles,
and concepts of it.
That is why, I expect help from you people. I have studied Filemon code,
and filespy code, and sfilter code, and hence decided to rewrite the
filemon tool myself from scratch, so that I could have a better grasp of
things.
Yes. But if there’s filter attached to a volume who returns FALSE from
FastIo calls, the driver receives only IRP based read/write requests.
Well, that might happen for third party drivers, if I know correctly,
all microsoft file system drivers, do support FastIo. Please correct me
if I am mistaken. More over, if a third party driver doesn’t support
FastIo, their performance would be so much degraded, that they better
use it instead, am I correct in this aspect?
Memory mapped files, I don’t think is in my current scope, and I don’t
think it is even possible to get these calls, unless we hook some
routines.
Am I correct in this?
No. You will see paging I/O reads or writes in case that the data are
not already
in the memory manager’s pool of mapped files.
But these paging Io calls would be from a different process context, not
the original application, correct? If this is true, there will be no way
to guess who the actual benifactor is. Or is there a way to figure that
out too?
Actually some guru once told me, that Filemon cannot see all file based
io. A much bette approach would be to hook Nt* calls in kernel mode
through SSDT hooking. This, he justified, would also take care of those
calls that are directed to the cache manager eventually. Is he correct
in this?
Please give your advices.
Once again, I thank you all for the help you extend.
L.
— Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17 You are currently subscribed
to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a
blank email to xxxxx@lists.osr.com
Bedanto
You should invest in the excellent driver and file systems courses.
You should remember all you have learned from the sfilter and filespy code;
and forget much of what
you have also learned from the filemon code.
You can not expect free help; you can just be grateful when it is given.
You have learned that paging i/o ocurs in arbitrary thread context; this is
correct.
You should not consider SSDT hooking; the “guru” gave you bad advice.
Kind regards
Lyndon
“Bedanto” wrote in message news:xxxxx@ntfsd…
>Fast I/O is an optional FSD interface and may not even be used for all I/O
>operations (for example, it is common to wait until the first I/O to >set
>up caching which is required for using fast I/O read or write.)
Tony, I am not sure whether I understand what you say. can you elaborate the
portion in the braces a bit more please, else give me some pointers, and I
think i can find out for myself.
>You must learn this informations anyway, it is essential
>for every filter/FS driver developer.
>If you are not able to search the information in the book,
>you are definitely not able to write drivers at all.
I am trying my best, but it is not easy to grasp these things, till you take
a few OSR classes or spend a few years looking at code. Unfortunately, the
option of attending these seminars are NULL for me.
>I respect you are newbie, but writing FS drivers or filters is not like
>writing a “hello, world” program. You have to learn the principles,
>and concepts of it.
That is why, I expect help from you people. I have studied Filemon code, and
filespy code, and sfilter code, and hence decided to rewrite the filemon
tool myself from scratch, so that I could have a better grasp of things.
>Yes. But if there’s filter attached to a volume who returns FALSE from
>FastIo calls, the driver receives only IRP based read/write requests.
Well, that might happen for third party drivers, if I know correctly, all
microsoft file system drivers, do support FastIo. Please correct me if I am
mistaken. More over, if a third party driver doesn’t support FastIo, their
performance would be so much degraded, that they better use it instead, am I
correct in this aspect?
> Memory mapped files, I don’t think is in my current scope, and I don’t
> think it is even possible to get these calls, unless we hook some
> routines.
> Am I correct in this?
>No. You will see paging I/O reads or writes in case that the data are not
>already
>in the memory manager’s pool of mapped files.
But these paging Io calls would be from a different process context, not the
original application, correct? If this is true, there will be no way to
guess who the actual benifactor is. Or is there a way to figure that out
too?
Actually some guru once told me, that Filemon cannot see all file based io.
A much bette approach would be to hook Nt* calls in kernel mode through SSDT
hooking. This, he justified, would also take care of those calls that are
directed to the cache manager eventually. Is he correct in this?
Please give your advices.
Once again, I thank you all for the help you extend.
L.