Hey I am also working on an encryption driver and I was just wondering why you dont handle BUFFERED_IO and DIRECT_IO in your completion routine? I think you will not receive many reads this way.
Hey,
Bufferd IO can also use the Irp->AssociatedIrp.SystemBuffer
not only the UserBuffer.
I think thats the point here!
>What about TopLevelIrp???
Hi Anton, several months ago in our last discussion you underestimate TopLevelIrp, and now you overestimate it )))
Once paging IO requests are supposed to be sent only by Cache Manager and Memory Manager
This is a preconception, who knows - may be you are a part. I wrote that it is better not to create paging Irp, but if there is no any choice - you can.
FSD may get “surprized” if TopLevelIrp on paging IO path is NULL.
NULL TopLevelIrp and paging IO is a normal situation. It is allowed by OS and FSD design. So, this surprises only you, but not any FSD or IO Manager.
Therefore, steps that you have described don’t seem to be sufficient
It is sufficient, trust me.
instead, you have to emulate top-level component’s request properly,
This is wrong.
i.e. to make FSD believe that a paging IO request has been originated by the Cache Manager or by Memory Manager…
You have a wrong notion of the TopLevelIrp meaning. I tried to explain its meaning during our last discussion, find it.
Slava Imameyev, xxxxx@hotmail.com
Slava,
Hi Anton, several months ago in our last discussion you underestimate
TopLevelIrp,
I did not…
Although you mentioned TopLevelIrp on the thread where we had a “disagreement”, I did not not argue with your statements about TopLevelIrp in any possible way. Instead, we argued whether you can take page faults on code on paging IO path to the mapped file - you were saying that you can take page faults on both code and data, and I was saying that you can take page faults on data but not on code…
and now you overestimate it )))
Actually, I don’t overestimate it - after all, I don’t say that FSD will *necessarily* bugcheck if it sees NULL TopLevelIrp on paging IO path, do I ? At the same time, I don’t see any reason why it cannot happen - FSD may do it simply to avoid possible data corruption in case if it decides that this situation is abnormal. Therefore, everything is up to FSD designers.
What I am saying is that if you want to make sure that your code works with any FSD you have to do thing in such way that FSD does not suspect any abnormality, no matter how cautious ( or even paranoid) its designers are…
You have a wrong notion of the TopLevelIrp meaning. I tried to explain its meaning
during our last discussion,
I am afraid you mistake me for someone else…
I remember the thread (although I did not participate in it) where you had a hard time explaining to the OP the meaning of TopLevelIrp - the bloke just refused to accept the fact that calling file functions when TopLevelIrp is not NULL is unsafe. I think this is exactly the thread that you mean here…
Anton Bassov
>Actually, I don’t overestimate it - after all,
At the same time, I don’t see any reason why it cannot happen - FSD may do it simply to avoid possible data corruption in case if it decides that this situation is abnormal.
… FSD does not suspect any abnormality,
You keep repeating the same mantra - “TopLevelIrp must be non NULL when FSD receives Paging IO request”. Do you believe in it? Do you think that this is a requirement? Would you please give the definite and concise answer - Yes or NO.
Slava Imameyev , xxxxx@hotmail.com
> You keep repeating the same mantra - "TopLevelIrp must be non NULL when FSD
receives Paging IO request". Do you believe in it? Do you think that this is a
requirement? Would you please give the definite and concise answer - Yes or NO.
Actually, it is not as simple as that…
Indeed, AFAIK, there is no official *requirement* saying that TopLevelIrp on paging IO path cannot be NULL (at least not that I am aware of), so that my “definite and concise answer” to your question is NO. However, as you have said yourself, MSFT “discourages” us from sending paging IRPs with IRP_PAGING_IO flag on our own initiative, which means that, as long as everyone follows MSFT “guidelines”, TopLevelIrp on paging IO path just cannot be NULL, because no one, apart from Memory Manager and Cache Manager, is going to do paging IO.
Therefore, overzealous FSD designer *may* take NULL TopLevelIrp on paging IO path as some kind of abnormality, and it is up to him/her to decide how to react to it…
To summarize, if I dealt with a problem that just does not have a solution other than sending paging IO request on your own initiative, I would emulate either Memory Manager’s or Cache Manager’s requests, rather than just sending IRP in an expectation that everything will work fine (even if all my experience shows that it does - after all, if your code works fine in 99.9% of cases, it is still not enough for saying that it is perfectly reliable)…
Anton Bassov
>so that my “definite and concise answer” to your question is NO.
Thank you, that is all I wanted.
–
Slava Imameyev, xxxxx@hotmail.com
wrote in message news:xxxxx@ntfsd…
>> You keep repeating the same mantra - “TopLevelIrp must be non NULL when
>> FSD
>> receives Paging IO request”. Do you believe in it? Do you think that this
>> is a
>> requirement? Would you please give the definite and concise answer - Yes
>> or NO.
>
> Actually, it is not as simple as that…
>
> Indeed, AFAIK, there is no official requirement saying that TopLevelIrp
> on paging IO path cannot be NULL (at least not that I am aware of), so
> that my “definite and concise answer” to your question is NO. However, as
> you have said yourself, MSFT “discourages” us from sending paging IRPs
> with IRP_PAGING_IO flag on our own initiative, which means that, as long
> as everyone follows MSFT “guidelines”, TopLevelIrp on paging IO path just
> cannot be NULL, because no one, apart from Memory Manager and Cache
> Manager, is going to do paging IO.
>
> Therefore, overzealous FSD designer may take NULL TopLevelIrp on paging
> IO path as some kind of abnormality, and it is up to him/her to decide how
> to react to it…
>
> To summarize, if I dealt with a problem that just does not have a solution
> other than sending paging IO request on your own initiative, I would
> emulate either Memory Manager’s or Cache Manager’s requests, rather than
> just sending IRP in an expectation that everything will work fine (even if
> all my experience shows that it does - after all, if your code works fine
> in 99.9% of cases, it is still not enough for saying that it is perfectly
> reliable)…
>
>
> Anton Bassov
>
> You keep repeating the same mantra - "TopLevelIrp must be non NULL when
FSD receives Paging IO request".
…which is just plain wrong, for instance, for registry hive flush operations
on XP and later.
–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Maxim,
> You keep repeating the same mantra - “TopLevelIrp must be non NULL when
> FSD receives Paging IO request”.
which is just plain wrong, for instance, for registry hive flush operations on XP and later.
Actually, I thought that the situation with the Registry flush stands as following:
-
W2K relies upon paged pool in order to keep it in memory. Therefore, when Registry gets flushed,
the paging IO goes to the paging file. -
XP and above rely upon file mapping functions. Therefore, when Registry gets flushed,
the paging IO goes to the mapped file, i.e. this is nothing more than a cache flush, so that Cache Manager is top-level component.
Am I wrong?
Anton Bassov
> 2. XP and above rely upon file mapping functions. Therefore, when Registry
gets flushed,
the paging IO goes to the mapped file, i.e. this is nothing more than a cache
flush, so that Cache Manager is top-level component.Am I wrong?
Yes.
On XP SP2+ (or maybe XP+), registry flushes come to the FSD with FCB locks
acquired and TopLevelIrp == NULL.
On NT4 and maybe newer OSes, FASTFAT has no AcquireForCcFlush routine, so, the
default routine is used which does not set TopLevelIrp, and once again - FCB
locks are acquired, and TopLevelIrp == NULL.
–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
In Narars book page 457:
...the FSD invokes a routine to determine whether the FSD can be the top-level component for the current IRP. The invoked routine is called SFsdlsIrpTopLevel(). This routine simply
checks the current value of the TopLevellrp field in the TLS to determine whether it has already been set. If the field contains a nonzero value, the FSD assumes that some other component is top-level for the current request; otherwise, the FSD sets the IRP pointer value in the TLS to indicate that the FSD itself is top-level for the current request... :~
In source code comment (misc.c- line 210)
* Function: SFsdIsIrpTopLevel()
*
* Description:
* Helps the FSD determine who the "top level" caller is for this
* request. A request can originate directly from a user process
* (in which case, the "top level" will be NULL when this routine
* is invoked), OR the user may have originated either from the NT
* Cache Manager/VMM ("top level" may be set), or this could be a
* recursion into our code in which we would have set the "top level"
* field the last time around.
Its based on Windows NT4 but....
Maxim,
On XP SP2+ (or maybe XP+), registry flushes come to the FSD with FCB locks
acquired and TopLevelIrp == NULL
How can this possibly happen??? If Cache Manager pre-acquires FCB locks before sending an IRP, TopLevelIrp is not going to be NULL. Please check FASTFAT sample and see how it implements Cache Manager’s callbacks - both operational and no-op callbacks set TopLevelIrp to FSRTL_CACHE_TOP_LEVEL_IRP (in fact, setting TopLevelIrp to FSRTL_CACHE_TOP_LEVEL_IRP is the only thing that no-op callbacks do). This behaviour is fully consistent with Nagar’s book…
Anton Bassov
> > On XP SP2+ (or maybe XP+), registry flushes come to the FSD with FCB
locks
> acquired and TopLevelIrp == NULLHow can this possibly happen???
I don’t know, but this happens a lot.
Registry flusher in XP (at least SP2) sends writes with TopLevelIrp == NULL and
FCB locks acquired, this is an experimental fact.
–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Maxim,
If registry flusher under XP relies upon file mapping functions, then the behaviour that you have described must,apparently, affect not only registry flushes but all memory-mapped files - after all, client code does not decide how flush will be handled by FSD and Cache Manager, does it?
Anton Bassov
I think that the flusher uses special code path in the file mapping
functions.
–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
wrote in message news:xxxxx@ntfsd…
> Maxim,
>
> If registry flusher under XP relies upon file mapping functions, then the
behaviour that you have described must,apparently, affect not only registry
flushes but all memory-mapped files - after all, client code does not decide
how flush will be handled by FSD and Cache Manager, does it?
>
>
> Anton Bassov
>
Maxim,
I think that the flusher uses special code path in the file mapping functions.
Even if there is some special path in file mapping functions, you still need to call FSD if you want to acquire FCB lock. Let’s say our system partition is formatted with FAT, rather than with NTFS ( once we have an access to FAT’s source, we can analyze the whole thing from FSD’s perspective). Do you see any way for flusher to acquire FCB locks without setting TopLevelIrp to FSRTL_CACHE_TOP_LEVEL_IRP, given that all Cache Manager’s callbacks set TopLevelIrp to FSRTL_CACHE_TOP_LEVEL_IRP, which can be established by looking at resrcup.c in FASTFAT sample???
Anton Bassov
>can analyze the whole thing from FSD’s perspective). Do you see any way for
flusher to acquire FCB locks without setting TopLevelIrp to
FSRTL_CACHE_TOP_LEVEL_IRP
Surely. The flusher can just access them thru FileObject->FsContext and take
them without touching TopLevelIrp.
–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Fair warning: it is a mistake to assume that you can just use the ERESOURCE objects in the FCB header. While that model worked at one point in the distant past, it is neither guaranteed nor required to work today. That is why we have the fast I/O functions and the FSRTL filter callbacks.
Assuming that you understand how a file systems internal locking works is a mistake, because it changes over time and is seldom as simple as you think it will be.
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
>>can analyze the whole thing from FSD’s perspective). Do you see any way for
>flusher to acquire FCB locks without setting TopLevelIrp to
>FSRTL_CACHE_TOP_LEVEL_IRP
Surely. The flusher can just access them thru FileObject->FsContext and take
them without touching TopLevelIrp.
Well, it just defeats the purpose of callbacks, don’t you think? Although technically it is possible to do so, I don’t (yet) see any reason why registry flusher may want to do that - after all, as far as Cache Manager and FSD are concerned, registry flush is just a usual file IO operation, so it does not seem to make sense (at least to me) to go for so convoluted solution at the time when straightforward one is available…
Anton Bassov
> Well, it just defeats the purpose of callbacks, don’t you think?
Maybe. Windows is full of fun.
–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com