Stream Context between multiple MiniFilter instances

On Thu, Sep 9, 2010 at 6:39 PM, Alex Carp wrote:
> Well, I think Aram’s implementation does not involve returning STATUS_REPARSE. IIRC he is switching the device stack.

Correct.

> However, in Vista and newer OSes, one can use ECPs to tag a CREATE even when STATUS_REPARSE is returned (so the new IRP_MJ_CREATE will have the ECP that the filter added attached to it)…

Nice, however I need to support Windows XP.

> One thing you might want to try is to call FltSetStreamContext from the post-Create callback on Instance1 using Instance2 as the key (this should be pretty easy since Instance1 already knows about Instance2 because it redirected the IO to it). Then on Instance2 you should be able to see the right context when calling FltGetStreamContext with Instance2 as the key, which I think is what you wanted…

Wow, never thought about that. Yes, I will try this approach and post
the results.

> Does this make sense ?

Yes, very much. Thanks.


Aram Hăvărneanu

On Thu, Sep 9, 2010 at 6:39 PM, Alex Carp wrote:
> Well, I think Aram’s implementation does not involve returning STATUS_REPARSE. IIRC he is switching the device stack.

Another question. If I change TargetInstance from Instance1 to
Instance2 in the CREATE path subsequent I/O will go directly to
Instance2.

However, if I pass TargetInstance unchanged in the CREATE path and
only change it in READ and WRITE paths, what will happen?

Thanks,


Aram Hăvărneanu

“However, if I pass TargetInstance unchanged in the CREATE path and only change it in READ and WRITE paths, what will happen?”

I think you might bugcheck the machine because the FO will be initialized by the file system on volume1 and then you will use that FO on volume2 (so whether it is a bugcheck or simply a failure depends entirely on the FS… however I think NTFS doesn’t like it and bugchecks when trying to interpret some fields in the FCB that are not there at all or are bogus.).

In general, a lot of the work in some types filters that either own FOs (SFO type filters and such) or that send requests from one volume to another (common in virtualization filters) is making sure that a FO never ends up on a file system that doesn’t own it (own = completed the IRP_MJ_CREATE). This is why, for example, such filters usually need to implement all possible operations (including name provider callbacks) even if they don’t care about the operation and they don’t change file names… Missing only one such path might result in the FO making its way to the wrong file system…

Thanks,
Alex.

Thanks for the valuable insights!


Aram Hăvărneanu

>> One thing you might want to try is to call FltSetStreamContext from the post-Create callback on Instance1 using Instance2 as the key (this should be pretty easy since Instance1 already knows about Instance2 because it redirected the IO to it). Then on Instance2 you should be able to see the right context when calling FltGetStreamContext with Instance2 as the key, which I think is what you wanted…

Wow, never thought about that. Yes, I will try this approach and post
the results.

As I need to create a new FILE_OBJECT in pre-create how can I make
sure I create the new FILE_OBJECT with the same access flags?


Aram Hăvărneanu

I’m not sure I follow you. Why do you need to create a new FILE_OBJECT in preCreate? I thought your minifilter was simply changing the TargetInstance to send it to a different volume…

Thanks,
Alex.

> I’m not sure I follow you. Why do you need to create a new FILE_OBJECT in preCreate? I thought your minifilter was simply changing the TargetInstance to send it to a different volume…

To quote from the docs: << A minifilter can change the value of the
TargetFileObject member. However, the new value must be a pointer to a
file object for a file that resides on the same volume as the instance
specified by the TargetInstance member. >>.

At pre-create the TargetFileObject is a pointer to a FILE_OBJECT that
references Instance1, instead of Instance2. From FILE_OBJECT
definition I see that it contains members such as DeviceObject which
references the wrong volume and Vpb which must be the volume parameter
block of the file on the target volume in order for caching to work
correctly. Because of this I understand I need to create a new
FILE_OBJECT that references the target volume on which Instance2
resides. Is my understanding wrong? Right now I do:

InitializeObjectAttributes(
&NewObjectAttributes,
&newFileName,
OBJ_OPENIF | OBJ_FORCE_ACCESS_CHECK,
NULL, NULL
);
status = FltCreateFileEx(
Globals.Filter, FltObjects->Instance,
&NewFileHandle, &NewFileObject,
Cbd->Iopb->Parameters.Create.SecurityContext->DesiredAccess,
&NewObjectAttributes,
&NewIoStatusBlock,
Cbd->Iopb->Parameters.Create.AllocationSize,
Cbd->Iopb->Parameters.Create.FileAttributes,
Cbd->Iopb->Parameters.Create.ShareAccess,
(Cbd->Iopb->Parameters.Create.Options & 0xFF000000) >> 24,
Cbd->Iopb->Parameters.Create.Options & 0x00FFFFFFFF,
Cbd->Iopb->Parameters.Create.EaBuffer,
Cbd->Iopb->Parameters.Create.EaLength,
IO_FORCE_ACCESS_CHECK
);

Thanks,


Aram Hăvărneanu

On Mon, Sep 20, 2010 at 3:11 AM, Aram Hăvărneanu wrote:
>> I’m not sure I follow you. Why do you need to create a new FILE_OBJECT in preCreate? I thought your minifilter was simply changing the TargetInstance to send it to a different volume…
>
> To quote from the docs: << A minifilter can change the value of the
> TargetFileObject member. However, the new value must be a pointer to a
> file object for a file that resides on the same volume as the instance
> specified by the TargetInstance member. >>.
>
> At pre-create the TargetFileObject is a pointer to a FILE_OBJECT that
> references Instance1, instead of Instance2. From FILE_OBJECT
> definition I see that it contains members such as DeviceObject which
> references the wrong volume and Vpb which must be the volume parameter
> block of the file on the target volume in order for caching to work
> correctly. Because of this I understand I need to create a new
> FILE_OBJECT that references the target volume on which Instance2
> resides. Is my understanding wrong? Right now I do:
>
> InitializeObjectAttributes(
> &NewObjectAttributes,
> &newFileName,
> OBJ_OPENIF | OBJ_FORCE_ACCESS_CHECK,
> NULL, NULL
> );
> status = FltCreateFileEx(
> Globals.Filter, FltObjects->Instance,
> &NewFileHandle, &NewFileObject,
> Cbd->Iopb->Parameters.Create.SecurityContext->DesiredAccess,
> &NewObjectAttributes,
> &NewIoStatusBlock,
> Cbd->Iopb->Parameters.Create.AllocationSize,
> Cbd->Iopb->Parameters.Create.FileAttributes,
> Cbd->Iopb->Parameters.Create.ShareAccess,
> (Cbd->Iopb->Parameters.Create.Options & 0xFF000000) >> 24,
> Cbd->Iopb->Parameters.Create.Options & 0x00FFFFFFFF,
> Cbd->Iopb->Parameters.Create.EaBuffer,
> Cbd->Iopb->Parameters.Create.EaLength,
> IO_FORCE_ACCESS_CHECK
> );

At the moment this doesn’t seem to work, the new file is opened in my
filter and I swap the FILE_OBJECT, but applications fail opening the
file in user mode. I use a modified simrep to do stack switching
instead of reparse and my debug output says things should work:

FsRdr: FsRdrPreCreate -> Processing create for file
\Device\HarddiskVolume2\x\y\ (Cbd = 86794F5C, FileObject = 867B3028)
FsRdr: FsRdrPreCreate -> Setting the callback data dirty flag.
FsRdr: FsRdrPreCreate -> Changed target for file \x\y. (Cbd =
86794F5C, NewFileObject = 86AF3288)
OpenedFileName = \Device\HarddiskVolume2\x\y<br> NewName = \Device\HarddiskVolume3\a\b<br>
As you can see, I have a new, different FILE_OBJECT and the new path
is the correct path (otherwise I couldn’t have opened the file in
kernel mode to obtain my FILE_OBJECT anyway). Userspace applications
fail:

W:&gt;dir x\y<br> Volume in drive W is Test
Volume Serial Number is 7496-0619

Directory of W:\x\y

File Not Found

Or Explorer:
W:\x\y refers to a location that is unavailable. It could be on a hard
drive on this computer, or on a network. Check to make sure that the
disk is properly inserted, or that you are connected to the Internet
or your network, and then try again. If it still cannot be located,
the information might have been moved to a different location.

What should I investigate further?

Thanks,


Aram Hăvărneanu

Yeah, I think you got it wrong… The FILE_OBJECT doesn’t yet reside on any volume. That happens when the FS receives the IRP_MJ_CREATE and finds the stream that the FILE_OBJECT is describing (via the FileName and RelatedFileObject members) and associates an SCB with it. So since you’re in preCreate this hasn’t yet happened.

Also, please consider what would happen if you actually went through with this. The FS on that volume would see two IRP_MJ_CREATEs with the same FILE_OBJECT, the second one with a FILE_OBJECT that’s actually in use… not sure what would happen, but I’m pretty sure it would be bad.

I’m sure I’ve asked this before, but why not use STATUS_REPARSE ? This model of redirecting creates using the TargetInstance is pretty flawed anyway…

Thanks,
Alex.

On Mon, Sep 20, 2010 at 4:28 AM, Alex Carp wrote:
> I’m sure I’ve asked this before, but why not use STATUS_REPARSE ? This model of redirecting creates using the TargetInstance is pretty flawed anyway…

Because I can’t track context with STATUS_REPARSE in pre-Vista OSes.


Aram Hăvărneanu

On Mon, Sep 20, 2010 at 4:28 AM, Alex Carp wrote:
> Yeah, I think you got it wrong… The FILE_OBJECT doesn’t yet reside on any volume. That happens when the FS receives the IRP_MJ_CREATE and finds the stream that the FILE_OBJECT is describing (via the FileName and RelatedFileObject members) and associates an SCB with it. So since you’re in preCreate this hasn’t yet happened.

That makes sense. I have tried this approach first, but I got the
exact same errors (see previous email) I get with my second approach.
Stuff seems to be happening correctly in the driver, but usermode
applications can’t open the file.


Aram Hăvărneanu

On Mon, Sep 20, 2010 at 4:35 AM, Aram Hăvărneanu wrote:
> On Mon, Sep 20, 2010 at 4:28 AM, Alex Carp wrote:
>> Yeah, I think you got it wrong… The FILE_OBJECT doesn’t yet reside on any volume. That happens when the FS receives the IRP_MJ_CREATE and finds the stream that the FILE_OBJECT is describing (via the FileName and RelatedFileObject members) and associates an SCB with it. So since you’re in preCreate this hasn’t yet happened.
>
> That makes sense. I have tried this approach first, but I got the
> exact same errors (see previous email) I get with my second approach.
> Stuff seems to be happening correctly in the driver, but usermode
> applications can’t open the file.

Actually I did another test, and in this first case the error would be
“The parameter is incorrect.” for both dir and explorer.exe.


Aram Hăvărneanu

>> I’m sure I’ve asked this before, but why not use STATUS_REPARSE  ? This model of redirecting creates using the TargetInstance is pretty flawed anyway…

Because I can’t track context with STATUS_REPARSE in pre-Vista OSes.

Since this doesn’t seem to work for some unknown reason and I need to
somehow mark' the redirected requests, I wonder if I can't add a stamp’ to the new file name I set when I return STATUS_REPARSE and
have a filter attached to the other volume remove this `stamp’. I’m
not sure how this works since the file name is cached and filters
attached to the other volume may get stamped requests. Can I
invalidate the cache somehow? I’m not sure if that’s safe because some
filters may be running in a context where it would not be safe to
re-enter the filesystem, though since I own the files I need to
process I guess that would not be a problem for me since I process
only regular files and skip paging I/O…

Is there a workaround?

Thanks,


Aram Hăvărneanu

I’ve been thinking about it as well and there doesn’t seem to be a good, reliable approach (which makes sense or MS wouldn’t have spent time working on ECPs if there was a better way).

The easiest thing is to do something to the file name that you can easily identify. However, any change to the actual name has a couple of serious issues:

  1. you might collide with someone else’s change depending on how you change the name (if you add a string to the end of the file for example then unless it is unique then someone else might use the same tag) - using a GUID for example would take care of this
  2. if you add a string then you might change the namespace… for example, if you add a GUID, then you will no longer allow for 32767 character paths, but rather 32759 ( sizeof(UUID) is 16, which is 8 unicode chars in windows ).
  3. if you add or change the string then filters above you might be unable to parse the path or to find it on disk or something like this, which might alter behavior of the IO stack…
  4. FltGetFileNameInformation would get confused…

Everything else seems just as bad.

I think this might be a good time to think about your requirements and see if there is a better solution… For example, it looks like you want to map a folder on a volume (V1) to a folder that lives on a different volume (V2) and not allow access to the folder through the volume it actually resides on (V2) but only through the folder on V1. Perhaps you could do something with ACLs? For example, one thing I think might work would be to restrict access to that folder and then before you reparse you could change the security context on the create (in Parameters.Create.SecurityContext) so that it can actually open the folder…

Would this work ?

Thanks,
Alex.

On Mon, Sep 20, 2010 at 10:17 PM, Alex Carp
wrote:
> 2. if you add a string then you might change the namespace… for example, if you add a GUID, then you will no longer allow for 32767 character paths, but rather 32759 ( sizeof(UUID) is 16, which is 8 unicode chars in windows ).

No problem, I won’t ADD a GUID to the file name, but simply USE a
GUID (generated randomly at runtime) for the whole file name and
manually maintain a mapping between GUIDs and real file names.

> 3. if you add or change the string then filters above you might be unable to parse the path or to find it on disk or something like this, which might alter behavior of the IO stack…

Absolutely correct.

> 4. FltGetFileNameInformation would get confused…

This is my main concern as well.

> I think this might be a good time to think about your requirements and see if there is a better solution… For example, it looks like you want to map a folder on a volume (V1) to a folder that lives on a different volume (V2) and not allow access to the folder through the volume it actually resides on (V2) but only through the folder on V1. Perhaps you could do something with ACLs? For example, one thing I think might work would be to restrict access to that folder and then before you reparse you could change the security context on the create (in Parameters.Create.SecurityContext) so that it can actually open the folder…
>
> Would this work ?

Wouldn’t work. The issue is not controlling access, but controlling
the operation of some filters that reside on V2. This filters do
logging and custom data processing that should not happen when the
file is accessed through V2, but only through V1. The file should be
accessible directly through V2, but the logging and the data
modification should not take place unless the access is through V1. I


Aram Hăvărneanu

Well, replacing the name with a GUID would work provided you also implement name provider callbacks that return the proper name. This might still impact legacy filters above yours though.

Thanks,
Alex.

On Mon, Sep 20, 2010 at 11:53 PM, Alex Carp
wrote:
> Well, replacing the name with a GUID would work provided you also implement name provider callbacks that return the proper name. This might still impact legacy filters above yours though.

Yes, the problem is minor (very unlikely to occur and would only have
impact if file is accessed through V1, while most access is through
V2) but I like to do things the proper way, in order to withstand the
test of time and not cause any problems for any filter.

Maybe I can create bogus temporary files on V2 so if a filter
intercepts a request and gets my unprocessed GUID would still have a
legal file to work with?


Aram Hăvărneanu

Well, that’s not usually the problem (that the filter doesn’t have a file to work with) but rather that a filer above yours might operate on the wrong file. I don’t know exactly where you sit in the altitude list, but assuming an antivirus is installed above your filter on V2, it will see the GUID file name in preCreate. If it tries to scan the file then it needs to get to the right one otherwise it might miss malicious files (if you create bogus files and not fix the names properly, then it might scan those). The same might be the case for an encryption filter that only encrypts .doc files… If they see the name as a GUID, then they might not know they need to decrypt the file…

There is yet another problem… In replacing the GUID with the actual name when the IRP_MJ_CREATE reaches your minifilter, you are effectively injecting that name in the middle of the IO stack. However, names belong at the top of the stack, because you have no guarantees that the namespace looks the same way at all levels. For example, imagine a minifilter above yours which implements a flat namespace on the volume consisting only of IDs and where it maintains the illusion of a hierarchical namespace by keeping track of ID to name mapping at its level. If this minifilter is above yours then the “\foo\bar.txt” file which might be exposed at the top of the stack simply doesn’t exist at your level… If you think this is too crazy of a scenario, keep in mind that in vista and newer OSes there is an inbox microsoft minifilter that does something pretty similar for some files (I’m talking about luafv.sys).

Now, I can’t tell whether these are minor or major problems (this is usually between you and your customers… perhaps their environment is locked down enough that there is never another filter loaded… )… All I can do is point out possible things to be concerned about.

Thanks,
Alex.

I have seen file-system virtualization filters that effectively strip off the ECPs when they notice a STATUS_REPARSE, so ECPs aren’t even a perfect solution on Windows 7. In general, its much easier if you only store advisory information in the ECP. For example, I had lengthy bit of processing to do prior to reparsing the create to the correct volume, so I cache that work in the ECP. If the ECP made it through the reparse handling, GREAT, if not, well, it slowed down the second part of the create. Perhaps this behavior is actually a bug in that filter… I dunno… maybe they did it on purpose for some other reason.

AFAIK, a create operation will be fully processed within a single thread context. If you only need to cache information, then you could have a global map (protected by a lock obviously) going from thread-id to “your information”. Then when you saw the request on the second volume, you could check to see if it was related to the per-thread cached request. This would still have some holes, because a program could … “open C:\A, then you reparse to D:\A, then some filter fails the request, then the app could open D:\A just because” … and then you wouldn’t have the correct behavior.

I saw a mention somewhere that pre-vista OS’s use the extended attributes fields to pass some information down for the filter manager (would have been done with ECPs on the newer OS). Maybe if you manipulate the EA list it will be propagated across the reparse on the older OS’s? Perhaps this is just a variant on what Alex proposed (manipulating the security contexts… some part of which will be propagated across reparses).

I think a filter that does that is broken. IO manager relies on those ECPs
being preserved in some scenarios.

You’re right that the thread context is the same for IO issued by the IO
manager. But there are no guarantees around it and in fact there are filters
that break this by pending IRP_MJ_CREATEs in preCreate and then completing
them in a different thread. So if you happen to be below such a filter, you
will see a different thread than the one where IO manager issued the create
at the top of the stack. And there is still the issue of the failed
STATUS_REPARSE and a new request coming in on the same thread (and that does
happen quite a lot in some environments… like SRV for example).

I don’t think EAs would work, because I think that the IO manager would
repopulate the EA buffer and EaLength from the open packet on each
subsequent CREATE operation. I’m pretty sure the original EA and EaLength
don’t get updated in a STATUS_REPARSE case, and even if they did, the IO
model has no mechanism by which updates to the IO_STACK_LOCATION get
propagated back up during IRP completion, meaning that setting them in the
current stack location will not propagate them up to the IO manager anyway.

I’m pretty sure the security context could be used in this manner. The thing
that makes this work (potentially) is that the security context is captured
early on (so subsequent reparses still happen in the same context) which
means it’s probably going to be unchanged throughout the call. Also, it is a
pointer to a buffer as opposed to a value, which means that changing
something in the buffer (but not the buffer itself) should be persistent
across calls. Still a hack though, but if this restricted to XP only (ECPs
are still the better option in Vista+) and if changes to the security
context still keep it consistent, then I think it has a fairly good chance
of success.

Thanks,
Alex.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@moka5.com
Sent: Tuesday, September 21, 2010 9:49 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Stream Context between multiple MiniFilter instances

I have seen file-system virtualization filters that effectively strip off
the ECPs when they notice a STATUS_REPARSE, so ECPs aren’t even a perfect
solution on Windows 7. In general, its much easier if you only store
advisory information in the ECP. For example, I had lengthy bit of
processing to do prior to reparsing the create to the correct volume, so I
cache that work in the ECP. If the ECP made it through the reparse
handling, GREAT, if not, well, it slowed down the second part of the create.
Perhaps this behavior is actually a bug in that filter… I dunno… maybe
they did it on purpose for some other reason.

AFAIK, a create operation will be fully processed within a single thread
context. If you only need to cache information, then you could have a
global map (protected by a lock obviously) going from thread-id to “your
information”. Then when you saw the request on the second volume, you could
check to see if it was related to the per-thread cached request. This would
still have some holes, because a program could … “open C:\A, then you
reparse to D:\A, then some filter fails the request, then the app could open
D:\A just because” … and then you wouldn’t have the correct behavior.

I saw a mention somewhere that pre-vista OS’s use the extended attributes
fields to pass some information down for the filter manager (would have been
done with ECPs on the newer OS). Maybe if you manipulate the EA list it
will be propagated across the reparse on the older OS’s? Perhaps this is
just a variant on what Alex proposed (manipulating the security contexts…
some part of which will be propagated across reparses).


NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars (including our new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer