async IO

amitr0-2 · July 3, 2006, 11:26am

IF I know correctly, IRP_MJ_CREATE is synchronous in nature (if teh call is
through IO manager). Keeping this in mind, I had designed some file name
based filters in my FSFD. The basic code is like this:

In create I fetch the name and store it away in a Hashed list based on the
FO. Now the filters are implemented based of file tyes. Eg, If I want to
mask off logs of DLLS being opened/read etc, I add this type of a filter and
those IRPs are bypassed and sent down to the lower driver.

Strange I find that read IRPs are bypassing the filters!!! I cannot
understand this, as while logging the IRPS I fetch the filename from the
HASH list and no where else!!!

can someone correct me in this issue…

–

amitr0

OSR_Community_User · July 4, 2006, 4:46am

“IF I know correctly, IRP_MJ_CREATE is synchronous in nature (if teh call is through IO manager). Keeping this in mind, I had designed some file name based filters in my FSFD.”

Yes, all CREATES are synchronous as far as I know.

"In create I fetch the name and store it away in a Hashed list based on the FO. Now the filters are implemented based of file tyes. Eg, If I want to mask off logs of DLLS being opened/read etc, I add this type of a filter and those IRPs are bypassed and sent down to the lower driver. "

Remember, sometimes the cache manager will create it’s own FO’s for a file - you won’t see a create for these fileobjects in your filter. If you can’t see these FO’s being created; obviously you can’t add them to your hash list. I would venture to say IoCreateStreamFileObject and IoCreateStreamFileObjectLite are the culprits here.

“Strange I find that read IRPs are bypassing the filters!!! I cannot understand this, as while logging the IRPS I fetch the filename from the HASH list and no where else!!!”

If my understanding is correct, this would cause this. The Cache manager’s private FO’s are not getting added to your hash list, since your file name look up is based upon the FO in your CREATE (the ones you can see). Thus, these reads are slippin’ by unchecked.

Just drinkin’ a beer or six,

m.

Slava_Imameyev · July 4, 2006, 6:45am

Hi, the IO Managet waits for the create IRP completion( which has the IRP_SYNCHRONOUS_API flag set ), but FSD drivers can return STATUS_PENDING and post the IRP in a worker queue.
As I know the Cache Manager never creates its own file objects using IoCreateStreamFileObject( it uses the FO sent by the FSD calling Cc…). FSDs can call IoCreateStreamFileObject to create FOs and initialize caching for this FOs.
And also you must remove the file object’s entry from the hash on close( IRP_MJ_CLOSE ) request.
“amitr0” wrote in message news:xxxxx@ntfsd…
IF I know correctly, IRP_MJ_CREATE is synchronous in nature (if teh call is through IO manager). Keeping this in mind, I had designed some file name based filters in my FSFD. The basic code is like this:

In create I fetch the name and store it away in a Hashed list based on the FO. Now the filters are implemented based of file tyes. Eg, If I want to mask off logs of DLLS being opened/read etc, I add this type of a filter and those IRPs are bypassed and sent down to the lower driver.

Strange I find that read IRPs are bypassing the filters!!! I cannot understand this, as while logging the IRPS I fetch the filename from the HASH list and no where else!!!

can someone correct me in this issue…

–

- amitr0

Slava_Imameyev · July 4, 2006, 6:58am

How can the Cache Manager initialize the file object? I think the FSD will be confused by the FO which it haven’t initialize.
wrote in message news:xxxxx@ntfsd…
“IF I know correctly, IRP_MJ_CREATE is synchronous in nature (if teh call is through IO manager). Keeping this in mind, I had designed some file name based filters in my FSFD.”

Yes, all CREATES are synchronous as far as I know.

"In create I fetch the name and store it away in a Hashed list based on the FO. Now the filters are implemented based of file tyes. Eg, If I want to mask off logs of DLLS being opened/read etc, I add this type of a filter and those IRPs are bypassed and sent down to the lower driver. "

Remember, sometimes the cache manager will create it’s own FO’s for a file - you won’t see a create for these fileobjects in your filter. If you can’t see these FO’s being created; obviously you can’t add them to your hash list. I would venture to say IoCreateStreamFileObject and IoCreateStreamFileObjectLite are the culprits here.

“Strange I find that read IRPs are bypassing the filters!!! I cannot understand this, as while logging the IRPS I fetch the filename from the HASH list and no where else!!!”

If my understanding is correct, this would cause this. The Cache manager’s private FO’s are not getting added to your hash list, since your file name look up is based upon the FO in your CREATE (the ones you can see). Thus, these reads are slippin’ by unchecked.

Just drinkin’ a beer or six,

m.

OSR_Community_User · July 4, 2006, 7:18am

Ok, the FSD creates these ‘shadow’ file objects… I do recall reading that the CC create’s it’s own FO’s sometimes - obviously with the assistance of the fsd.

Perhaps you missed that last line of my email, "
Just drinkin’ a beer or six,". Never the less, I think in ‘theory’ I’m correct regarding FOs he has never seen and never added to his hash are the issue…

Then again, I’m usually wrong…

m

How can the Cache Manager initialize the file object? I think the FSD will be confused by the FO which it haven’t initialize.
wrote in message news:xxxxx@ntfsd…
“IF I know correctly, IRP_MJ_CREATE is synchronous in nature (if teh call is through IO manager). Keeping this in mind, I had designed some file name based filters in my FSFD.”

Yes, all CREATES are synchronous as far as I know.

"In create I fetch the name and store it away in a Hashed list based on the FO. Now the filters are implemented based of file tyes. Eg, If I want to mask off logs of DLLS being opened/read etc, I add this type of a filter and those IRPs are bypassed and sent down to the lower driver. "

Remember, sometimes the cache manager will create it’s own FO’s for a file - you won’t see a create for these fileobjects in your filter. If you can’t see these FO’s being created; obviously you can’t add them to your hash list. I would venture to say IoCreateStreamFileObject and IoCreateStreamFileObjectLite are the culprits here.

“Strange I find that read IRPs are bypassing the filters!!! I cannot understand this, as while logging the IRPS I fetch the filename from the HASH list and no where else!!!”

If my understanding is correct, this would cause this. The Cache manager’s private FO’s are not getting added to your hash list, since your file name look up is based upon the FO in your CREATE (the ones you can see). Thus, these reads are slippin’ by unchecked.

Just drinkin’ a beer or six,

m.

—

Baja_Mani · July 5, 2006, 2:11am

FOs not seen and added to the hash is the issue. That’s okay. But are there
any FOs which a fs filter will not see?
I thought that a fs filter must never miss any FO created (of course it
can’t see the ones created before it was loaded).
Even if CC creates the FO with the help of fsd, the fs filter must obvisouly
be able to see it.

wrote in message news:xxxxx@ntfsd…
Ok, the FSD creates these ‘shadow’ file objects… I do recall reading that
the CC create’s it’s own FO’s sometimes - obviously with the assistance of
the fsd.

Perhaps you missed that last line of my email, “
Just drinkin’ a beer or six,”. Never the less, I think in ‘theory’ I’m
correct regarding FOs he has never seen and never added to his hash are the
issue…

Then again, I’m usually wrong…

m

How can the Cache Manager initialize the file object? I think the FSD will
be confused by the FO which it haven’t initialize.
wrote in message news:xxxxx@ntfsd…
“IF I know correctly, IRP_MJ_CREATE is synchronous in nature (if teh call is
through IO manager). Keeping this in mind, I had designed some file name
based filters in my FSFD.”

Yes, all CREATES are synchronous as far as I know.

"In create I fetch the name and store it away in a Hashed list based on the
FO. Now the filters are implemented based of file tyes. Eg, If I want to
mask off logs of DLLS being opened/read etc, I add this type of a filter and
those IRPs are bypassed and sent down to the lower driver. "

Remember, sometimes the cache manager will create it’s own FO’s for a file -
you won’t see a create for these fileobjects in your filter. If you can’t
see these FO’s being created; obviously you can’t add them to your hash
list. I would venture to say IoCreateStreamFileObject and
IoCreateStreamFileObjectLite are the culprits here.

“Strange I find that read IRPs are bypassing the filters!!! I cannot
understand this, as while logging the IRPS I fetch the filename from the
HASH list and no where else!!!”

If my understanding is correct, this would cause this. The Cache manager’s
private FO’s are not getting added to your hash list, since your file name
look up is based upon the FO in your CREATE (the ones you can see). Thus,
these reads are slippin’ by unchecked.

Just drinkin’ a beer or six,

m.

—

OSR_Community_User · July 5, 2006, 2:29am

Read this article about stream tracking, this is what I was recalling last night when speaking of FOs he wouldn’t see…

http://www.osronline.com/article.cfm?id=102

m

Slava_Imameyev · July 5, 2006, 7:05am

Search the archive, we read and discussed this article many times, especially this quote
“For each IRP_MJ_CLOSE the filter driver decrements the reference count on the per-file context structure if the FO_STREAM_FILE bit is not set for the file object and if CcGetFileObjectFromSectionPtrs does not return this file object.”

And this article says nothing about " the CC create’s it’s own FO’s sometimes - obviously with the assistance of the fsd".
wrote in message news:xxxxx@ntfsd…
Read this article about stream tracking, this is what I was recalling last night when speaking of FOs he wouldn’t see…

http://www.osronline.com/article.cfm?id=102

m

OSR_Community_User · July 5, 2006, 7:55am

Slava,

I was responding to Mani with the link. He asked, “But are there any FOs which a fs filter will not see?”

In the article I posted, it states;

A file system has several mechanisms for creating file objects that it may use for its own internal processing. These include IoCreateStreamFileObject, IoCreateStreamFileObjectLite, and IoCreateStreamFileObjectEx. Not all of these are available in all versions, but even the Windows NT 4.0 IFS Kit contained the IoCreateStreamFileObject call. Both IoCreateStreamFileObject and IoCreateStreamFileObjectLite are documented in the Windows XP IFS Kit and each provides similar functionality – they create file objects based upon either an existing file object or device object. The IFS Kit documentation sums up the problem for file system filter drivers, in the page for IoCreateStreamFileObjectLite:

“Thus filter drivers should expect to receive IRP_MJ_CLOSE requests for previously unseen file objects.”

m

Search the archive, we read and discussed this article many times, especially this quote
“For each IRP_MJ_CLOSE the filter driver decrements the reference count on the per-file context structure if the FO_STREAM_FILE bit is not set for the file object and if CcGetFileObjectFromSectionPtrs does not return this file object.”

And this article says nothing about " the CC create’s it’s own FO’s sometimes - obviously with the assistance of the fsd".

wrote in message news:xxxxx@ntfsd…
Read this article about stream tracking, this is what I was recalling last night when speaking of FOs he wouldn’t see…

http://www.osronline.com/article.cfm?id=102

m

—
Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

OSR_Community_User · July 5, 2006, 8:22am

As if this isn’t already complex enough, we have found some products
that explicitly skip over your filter - sometimes. For example, we
recently found a product that sent an IRP_MJ_CREATE through our filter
but then send the final IRP_MJ_CLOSE directly to the underlying file
system. Needless to say, we were not very happy when we saw a new
IRP_MJ_CREATE against a file object we still thought was “alive”.

So, the interactions we discuss here are those of the base OS and “all
bets are off” when you start working with third party components.

Regards,

Tony

Tony Mason

Consulting Partner

OSR Open Systems Resources, Inc.

http://www.osr.com

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@comcast.net
Sent: Wednesday, July 05, 2006 7:55 AM
To: ntfsd redirect
Subject: [ntfsd] Re:async IO

Slava,

I was responding to Mani with the link. He asked, “But are there any FOs
which a fs filter will not see?”

In the article I posted, it states;

A file system has several mechanisms for creating file objects that it
may use for its own internal processing. These include
IoCreateStreamFileObject, IoCreateStreamFileObjectLite, and
IoCreateStreamFileObjectEx. Not all of these are available in all
versions, but even the Windows NT 4.0 IFS Kit contained the
IoCreateStreamFileObject call. Both IoCreateStreamFileObject and
IoCreateStreamFileObjectLite are documented in the Windows XP IFS Kit
and each provides similar functionality - they creat e file objects
based upon either an existing file object or device object. The IFS Kit
documentation sums up the problem for file system filter drivers, in the
page for IoCreateStreamFileObjectLite:

“Thus filter drivers should expect to receive IRP_MJ_CLOSE requests for
previously unseen file objects.”

m

Search the archive, we read and discussed this article many times,
especially this quote

“For each IRP_MJ_CLOSE the filter driver decrements the reference count
on the per-file context structure if the FO_STREAM_FILE bit is not set
for the file object and if CcGetFileObjectFromSectionPtrs does not
return this file object.”

And this article says nothing about " the CC create’s it’s own FO’s
sometimes - obviously with the assistance of the fsd".

wrote in message news:xxxxx@ntfsd…

Read this article about stream tracking, this is what I was
recalling last night when speaking of FOs he wouldn’t see…

http://www.osronline.com/article.cfm?id=102

m

—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

OSR_Community_User · July 7, 2006, 9:16am

Tony,
Have you or anyone else ever considered creating a list of drivers that don’t play by the rules?
A list of these products/drivers would be helpful to the community in design and trouble shooting, and perhaps, if these companies that broke the rules were ‘called out’, they would design their drivers with a little more care and thought…
If OSR went ahead with something like this, as long as it was factual these companies would be legally neutered.
m.

As if this isn’t already complex enough, we have found some products that explicitly skip over your filter - sometimes. For example, we recently found a product that sent an IRP_MJ_CREATE through our filter but then send the final IRP_MJ_CLOSE directly to the underlying file system. Needless to say, we were not very happy when we saw a new IRP_MJ_CREATE against a file object we still thought was “alive”.

So, the interactions we discuss here are those of the base OS and “all bets are off” when you start working with third party components.

Regards,

Tony