Hello All
In my filter driver I am trying to monitor all the requests with
IRP_PAGING_IO, IRP_NOCACHE and IRP_SYNCHRONOUS_PAGING_IO. For some reason,
when I open an existing file from an application, the filter driver does not
see any READ requests with these flags (only the usual read requests).
However, the file content is delivered to the application correctly. Is it
possible that the data is read without using any of the non-cached requests?
Thanks,
Gregory
Absolutely. The read is cached from the application, the data is
already in the cache and thus you do not see any non-cached operations
because they aren’t necessary.
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
Thanks, Tony.
I went through the archives of this list and noticed that the usual
recommendation for encryption drivers was to filter non-cached IOs only. So
I assumed that all the data is at some time read/written from/to the disk
using non-cached IOs. Now I’m trying to understand what are the cases when
this assumption is wrong. From Rajeev Nagar’s book I understood that the
file system drivers usually first try to read the data from the cache,
causing a page fault if the data is not there. Now it appears that sometimes
a shortcut is taken. Is there a way to recognize a case when the file system
will skip the cache? My first guess is that I can check in the FILE_OBJECT
whether the cache is initialized for the file. Is this correct?
Thanks a lot,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Tuesday, February 20, 2007 8:35 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
Absolutely. The read is cached from the application, the data is already in
the cache and thus you do not see any non-cached operations because they
aren’t necessary.
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
Non-cached is not necessarily the same as paging IO. Non-cached is requested to not be cached, while paging IO is the IO that will read/write the data to the media from the cache. I guess I will try to break it out, but this is complex and easily confused.
- CreateFile() creates context info in the file system driver. It might also initiate reads via the cache as the OS desires, but who cares?
- ReadFile() goes to the FSD which then sends it to the CacheManager. When the CacheManager has the data it returns.
- IRP_MJ_READ goes to the FSD from the CacheManager and then down the storage stack where the data is located. This is where you decrypt unless the file was opened with no buffering. Some FSDs can in some cases never honor the no buffering request. NTFS and compressed data is one case.
In #2 above, the ReadFile() can arrive as a FastIo or an IRP_MJ_READ. If a file has been opened before and the FSD and CacheManager knows it, then the CacheManager can provide the data without any more storage stack IOs. If it was decrypted before by a file system filter driver (FSFD) it will remain decrypted in the cache.
Why do you care if the file is cached or not? You only care if the request is going to the storage. There are flags in the IRPs that indicate they will go there. I think there are three bits, but I forget since I don’t do encryption any more.
“Gregory Dardyk” wrote in message news:xxxxx@ntfsd…
Thanks, Tony.
I went through the archives of this list and noticed that the usual recommendation for encryption drivers was to filter non-cached IOs only. So I assumed that all the data is at some time read/written from/to the disk using non-cached IOs. Now I’m trying to understand what are the cases when this assumption is wrong. From Rajeev Nagar’s book I understood that the file system drivers usually first try to read the data from the cache, causing a page fault if the data is not there. Now it appears that sometimes a shortcut is taken. Is there a way to recognize a case when the file system will skip the cache? My first guess is that I can check in the FILE_OBJECT whether the cache is initialized for the file. Is this correct?
Thanks a lot,
Gregory
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Tuesday, February 20, 2007 8:35 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
Absolutely. The read is cached from the application, the data is already in the cache and thus you do not see any non-cached operations because they aren’t necessary.
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
—
Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
David,
Thanks for your reply. The three bits you are talking about must be
IRP_PAGING_IO, IRP_NOCACHE and IRP_SYNCHRONOUS_PAGING_IO that I’ve listed in
the original post. These are exactly the requests that I am filtering in my
driver. The problem is that I never get a READ with any of these flags,
neither during CreateFile() nor during ReadFile(). I cannot think of a way
to load the data from the storage device to the cache without sending a READ
request with one of the three flags.
Thanks,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of David Craig
Sent: Wednesday, February 21, 2007 9:12 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Missing READ IOs in filter driver
Non-cached is not necessarily the same as paging IO. Non-cached is
requested to not be cached, while paging IO is the IO that will read/write
the data to the media from the cache. I guess I will try to break it out,
but this is complex and easily confused.
-
CreateFile() creates context info in the file system driver. It might
also initiate reads via the cache as the OS desires, but who cares?
-
ReadFile() goes to the FSD which then sends it to the CacheManager.
When the CacheManager has the data it returns.
-
IRP_MJ_READ goes to the FSD from the CacheManager and then down the
storage stack where the data is located. This is where you decrypt unless
the file was opened with no buffering. Some FSDs can in some cases never
honor the no buffering request. NTFS and compressed data is one case.
In #2 above, the ReadFile() can arrive as a FastIo or an IRP_MJ_READ. If a
file has been opened before and the FSD and CacheManager knows it, then the
CacheManager can provide the data without any more storage stack IOs. If it
was decrypted before by a file system filter driver (FSFD) it will remain
decrypted in the cache.
Why do you care if the file is cached or not? You only care if the request
is going to the storage. There are flags in the IRPs that indicate they
will go there. I think there are three bits, but I forget since I don’t do
encryption any more.
“Gregory Dardyk” wrote in message news:xxxxx@ntfsd…
Thanks, Tony.
I went through the archives of this list and noticed that the usual
recommendation for encryption drivers was to filter non-cached IOs only. So
I assumed that all the data is at some time read/written from/to the disk
using non-cached IOs. Now I’m trying to understand what are the cases when
this assumption is wrong. From Rajeev Nagar’s book I understood that the
file system drivers usually first try to read the data from the cache,
causing a page fault if the data is not there. Now it appears that sometimes
a shortcut is taken. Is there a way to recognize a case when the file system
will skip the cache? My first guess is that I can check in the FILE_OBJECT
whether the cache is initialized for the file. Is this correct?
Thanks a lot,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Tuesday, February 20, 2007 8:35 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
Absolutely. The read is cached from the application, the data is already in
the cache and thus you do not see any non-cached operations because they
aren’t necessary.
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
NTFS uses stream file object to handle paging I/O. For this FO
there are no IRP_MJ_CREATE request. This FO is created with
IoCreateStreamFileObject/IoCreateStreamFileObjectLite. For original FO and
stream FO uses the same fscontext. So you must save fscontext for the
original FO in IRP_MJ_CREATE to be able to determine paging I/O for that
file with stream file object.
Andrey Gunko
soft Xpansion Ukraine Ltd.
Programmer
Powered by eKnow-how
Artjoma St. 118B … 83048 Donetsk … Tel/Fax: +38 062 3818874 …
Internet: [http: www.soft-xpansion.com]
_____
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gregory Dardyk
Sent: Wednesday, February 21, 2007 9:37 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
David,
Thanks for your reply. The three bits you are talking about must be
IRP_PAGING_IO, IRP_NOCACHE and IRP_SYNCHRONOUS_PAGING_IO that I’ve listed in
the original post. These are exactly the requests that I am filtering in my
driver. The problem is that I never get a READ with any of these flags,
neither during CreateFile() nor during ReadFile(). I cannot think of a way
to load the data from the storage device to the cache without sending a READ
request with one of the three flags.
Thanks,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of David Craig
Sent: Wednesday, February 21, 2007 9:12 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Missing READ IOs in filter driver
Non-cached is not necessarily the same as paging IO. Non-cached is
requested to not be cached, while paging IO is the IO that will read/write
the data to the media from the cache. I guess I will try to break it out,
but this is complex and easily confused.
1. CreateFile() creates context info in the file system driver. It might
also initiate reads via the cache as the OS desires, but who cares?
2. ReadFile() goes to the FSD which then sends it to the CacheManager.
When the CacheManager has the data it returns.
3. IRP_MJ_READ goes to the FSD from the CacheManager and then down the
storage stack where the data is located. This is where you decrypt unless
the file was opened with no buffering. Some FSDs can in some cases never
honor the no buffering request. NTFS and compressed data is one case.
In #2 above, the ReadFile() can arrive as a FastIo or an IRP_MJ_READ. If a
file has been opened before and the FSD and CacheManager knows it, then the
CacheManager can provide the data without any more storage stack IOs. If it
was decrypted before by a file system filter driver (FSFD) it will remain
decrypted in the cache.
Why do you care if the file is cached or not? You only care if the request
is going to the storage. There are flags in the IRPs that indicate they
will go there. I think there are three bits, but I forget since I don’t do
encryption any more.
“Gregory Dardyk” wrote in message news:xxxxx@ntfsd…
Thanks, Tony.
I went through the archives of this list and noticed that the usual
recommendation for encryption drivers was to filter non-cached IOs only. So
I assumed that all the data is at some time read/written from/to the disk
using non-cached IOs. Now I’m trying to understand what are the cases when
this assumption is wrong. From Rajeev Nagar’s book I understood that the
file system drivers usually first try to read the data from the cache,
causing a page fault if the data is not there. Now it appears that sometimes
a shortcut is taken. Is there a way to recognize a case when the file system
will skip the cache? My first guess is that I can check in the FILE_OBJECT
whether the cache is initialized for the file. Is this correct?
Thanks a lot,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Tuesday, February 20, 2007 8:35 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
Absolutely. The read is cached from the application, the data is already in
the cache and thus you do not see any non-cached operations because they
aren’t necessary.
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com</http:>
Andrey,
Thanks for the tip. That’s exactly what my driver does. The question now is
whether I missed something and it is possible to read data without paging
IOs or I got everything right and there is just a bug in the code.
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gunko Andrey
Sent: Wednesday, February 21, 2007 9:54 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
NTFS uses stream file object to handle paging I/O. For this FO
there are no IRP_MJ_CREATE request. This FO is created with
IoCreateStreamFileObject/IoCreateStreamFileObjectLite. For original FO and
stream FO uses the same fscontext. So you must save fscontext for the
original FO in IRP_MJ_CREATE to be able to determine paging I/O for that
file with stream file object.
Andrey Gunko
soft Xpansion Ukraine Ltd.
Programmer
Powered by eKnow-how
Artjoma St. 118B … 83048 Donetsk … Tel/Fax: +38 062 3818874 …
Internet: [http: www.soft-xpansion.com]
_____
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gregory Dardyk
Sent: Wednesday, February 21, 2007 9:37 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
David,
Thanks for your reply. The three bits you are talking about must be
IRP_PAGING_IO, IRP_NOCACHE and IRP_SYNCHRONOUS_PAGING_IO that I’ve listed in
the original post. These are exactly the requests that I am filtering in my
driver. The problem is that I never get a READ with any of these flags,
neither during CreateFile() nor during ReadFile(). I cannot think of a way
to load the data from the storage device to the cache without sending a READ
request with one of the three flags.
Thanks,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of David Craig
Sent: Wednesday, February 21, 2007 9:12 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Missing READ IOs in filter driver
Non-cached is not necessarily the same as paging IO. Non-cached is
requested to not be cached, while paging IO is the IO that will read/write
the data to the media from the cache. I guess I will try to break it out,
but this is complex and easily confused.
1. CreateFile() creates context info in the file system driver. It might
also initiate reads via the cache as the OS desires, but who cares?
2. ReadFile() goes to the FSD which then sends it to the CacheManager.
When the CacheManager has the data it returns.
3. IRP_MJ_READ goes to the FSD from the CacheManager and then down the
storage stack where the data is located. This is where you decrypt unless
the file was opened with no buffering. Some FSDs can in some cases never
honor the no buffering request. NTFS and compressed data is one case.
In #2 above, the ReadFile() can arrive as a FastIo or an IRP_MJ_READ. If a
file has been opened before and the FSD and CacheManager knows it, then the
CacheManager can provide the data without any more storage stack IOs. If it
was decrypted before by a file system filter driver (FSFD) it will remain
decrypted in the cache.
Why do you care if the file is cached or not? You only care if the request
is going to the storage. There are flags in the IRPs that indicate they
will go there. I think there are three bits, but I forget since I don’t do
encryption any more.
“Gregory Dardyk” wrote in message news:xxxxx@ntfsd…
Thanks, Tony.
I went through the archives of this list and noticed that the usual
recommendation for encryption drivers was to filter non-cached IOs only. So
I assumed that all the data is at some time read/written from/to the disk
using non-cached IOs. Now I’m trying to understand what are the cases when
this assumption is wrong. From Rajeev Nagar’s book I understood that the
file system drivers usually first try to read the data from the cache,
causing a page fault if the data is not there. Now it appears that sometimes
a shortcut is taken. Is there a way to recognize a case when the file system
will skip the cache? My first guess is that I can check in the FILE_OBJECT
whether the cache is initialized for the file. Is this correct?
Thanks a lot,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Tuesday, February 20, 2007 8:35 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
Absolutely. The read is cached from the application, the data is already in
the cache and thus you do not see any non-cached operations because they
aren’t necessary.
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com</http:>
One stupid question: where are you watching those flags? The right place is
Irp->Flags, the pointer to irp stack not the current stack location.
Andrey Gunko
soft Xpansion Ukraine Ltd.
Programmer
Powered by eKnow-how
Artjoma St. 118B … 83048 Donetsk … Tel/Fax: +38 062 3818874 …
Internet: [http: www.soft-xpansion.com]
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gregory Dardyk
Sent: Wednesday, February 21, 2007 10:18 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
Andrey,
Thanks for the tip. That’s exactly what my driver does. The question now is
whether I missed something and it is possible to read data without paging
IOs or I got everything right and there is just a bug in the code.
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gunko Andrey
Sent: Wednesday, February 21, 2007 9:54 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
NTFS uses stream file object to handle paging I/O. For this FO
there are no IRP_MJ_CREATE request. This FO is created with
IoCreateStreamFileObject/IoCreateStreamFileObjectLite. For original FO and
stream FO uses the same fscontext. So you must save fscontext for the
original FO in IRP_MJ_CREATE to be able to determine paging I/O for that
file with stream file object.
Andrey Gunko
soft Xpansion Ukraine Ltd.
Programmer
Powered by eKnow-how
Artjoma St. 118B … 83048 Donetsk … Tel/Fax: +38 062 3818874 …
Internet: [http: www.soft-xpansion.com]
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gregory Dardyk
Sent: Wednesday, February 21, 2007 9:37 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
David,
Thanks for your reply. The three bits you are talking about must be
IRP_PAGING_IO, IRP_NOCACHE and IRP_SYNCHRONOUS_PAGING_IO that I’ve listed in
the original post. These are exactly the requests that I am filtering in my
driver. The problem is that I never get a READ with any of these flags,
neither during CreateFile() nor during ReadFile(). I cannot think of a way
to load the data from the storage device to the cache without sending a READ
request with one of the three flags.
Thanks,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of David Craig
Sent: Wednesday, February 21, 2007 9:12 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Missing READ IOs in filter driver
Non-cached is not necessarily the same as paging IO. Non-cached is
requested to not be cached, while paging IO is the IO that will read/write
the data to the media from the cache. I guess I will try to break it out,
but this is complex and easily confused.
1. CreateFile() creates context info in the file system driver. It might
also initiate reads via the cache as the OS desires, but who cares?
2. ReadFile() goes to the FSD which then sends it to the CacheManager.
When the CacheManager has the data it returns.
3. IRP_MJ_READ goes to the FSD from the CacheManager and then down the
storage stack where the data is located. This is where you decrypt unless
the file was opened with no buffering. Some FSDs can in some cases never
honor the no buffering request. NTFS and compressed data is one case.
In #2 above, the ReadFile() can arrive as a FastIo or an IRP_MJ_READ. If a
file has been opened before and the FSD and CacheManager knows it, then the
CacheManager can provide the data without any more storage stack IOs. If it
was decrypted before by a file system filter driver (FSFD) it will remain
decrypted in the cache.
Why do you care if the file is cached or not? You only care if the request
is going to the storage. There are flags in the IRPs that indicate they
will go there. I think there are three bits, but I forget since I don’t do
encryption any more.
“Gregory Dardyk” wrote in message news:xxxxx@ntfsd…
Thanks, Tony.
I went through the archives of this list and noticed that the usual
recommendation for encryption drivers was to filter non-cached IOs only. So
I assumed that all the data is at some time read/written from/to the disk
using non-cached IOs. Now I’m trying to understand what are the cases when
this assumption is wrong. From Rajeev Nagar’s book I understood that the
file system drivers usually first try to read the data from the cache,
causing a page fault if the data is not there. Now it appears that sometimes
a shortcut is taken. Is there a way to recognize a case when the file system
will skip the cache? My first guess is that I can check in the FILE_OBJECT
whether the cache is initialized for the file. Is this correct?
Thanks a lot,
Gregory
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Tuesday, February 20, 2007 8:35 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing READ IOs in filter driver
Absolutely. The read is cached from the application, the data is already in
the cache and thus you do not see any non-cached operations because they
aren’t necessary.
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com</http:></http:>