FS Filter Driver question

NTDEV
1

Dear NTDev folks,

We are writing a FileSystem Filter Driver, that mangles the contents of
a particular file, such that:

  1. the content is Mangled on Write, and
  2. Unmangled on Read.

Our Mangling actually increases the size of the file, and we also insert
our own header data in the beginning of the file.

However we still want to present the “correct” file-length to the client
programs (e.g. they shouldn’t know that the data is actually mangled
before it’s stored in the file).

We have been able to do this, to some degree, by handling the Read/Write
IRPs, and modifying the length fields in the QueryInformation IRPs (for
both File Information, and Directory Listings).

However, we are worried that interactions between the underlying
FileSystem Driver, and the Cache Manager may expose the real length of
the file in some cases (e.g. through the FileSize field in the
FSRTL_FCB_COMMON_HEADER structure stored in the FsContext field of the
FileObject), or cause other problems. We are not able to find a good
discussion on Data Modifying filter drivers in the IFS kit documentation
or in the “Windows NT File System Internals” Book by Rajeev Nagar.

FileSystem Filter Drivers that do non-length-preserving Encryption or
Compression must face the same issues that we are coming across. We
would really appreciate it if you folks could shed some light on how the
cache manager may affect FileSystem Filter Drivers that mangle the
content of the file such that the actual file length changes, or if you
could suggest some resources where this information is available.

Thanks in Advance.
-Sincerely,
-Anurag

2

Why put the “header” at the beginning? It is the most difficult place to put it and maintain any semblance of obfuscation. Put it at the end and give yourself enough space to permit expanding it easily. I would recommend that the last 128, 256, 512, etc bytes be the “header/trailer”. If expansion becomes required later, you can expand downwards from that fixed part of the header that will let you know the file is yours.

The following are questions you need to answer before you design your solution:

  1. Can the file be accessed in “mangled” form by any program, at any time? Backup?
  2. Can the file be modified by any of the Microsoft Office programs?
  3. Why do you care if one program “knows” the file size if wrong? What can be revealed?

----- Original Message -----
From: Anurag Sharma
To: NT Developers Interest List
Sent: Monday, January 13, 2003 8:46 PM
Subject: [ntdev] FS Filter Driver question

Dear NTDev folks,

We are writing a FileSystem Filter Driver, that mangles the contents of a particular file, such that:

  1. the content is Mangled on Write, and

  2. Unmangled on Read.

Our Mangling actually increases the size of the file, and we also insert our own header data in the beginning of the file.

However we still want to present the “correct” file-length to the client programs (e.g. they shouldn’t know that the data is actually mangled before it’s stored in the file).

We have been able to do this, to some degree, by handling the Read/Write IRPs, and modifying the length fields in the QueryInformation IRPs (for both File Information, and Directory Listings).

However, we are worried that interactions between the underlying FileSystem Driver, and the Cache Manager may expose the real length of the file in some cases (e.g. through the FileSize field in the FSRTL_FCB_COMMON_HEADER structure stored in the FsContext field of the FileObject), or cause other problems. We are not able to find a good discussion on Data Modifying filter drivers in the IFS kit documentation or in the “Windows NT File System Internals” Book by Rajeev Nagar.

FileSystem Filter Drivers that do non-length-preserving Encryption or Compression must face the same issues that we are coming across. We would really appreciate it if you folks could shed some light on how the cache manager may affect FileSystem Filter Drivers that mangle the content of the file such that the actual file length changes, or if you could suggest some resources where this information is available.

Thanks in Advance.

-Sincerely,

-Anurag

You are currently subscribed to ntdev as: xxxxx@yoshimuni.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

3

I don’t see any problem with FCB’s FileSize having bigger value than what
you report through other interfaces.
However, I must agreed with David that having header at the beginning of
the file is not the best solution. The only significant advantage
I can see is that you simplify handling of file expansion (only
considering that header has fixed size). However you gain a lot of
complications
trying to keep the header out of the cache, dealing with
FileObject->CurrentByteOffset for sequential files (these are just from
the top of my head).
So, unless there are particular reasons why you want to have your header
at the beginning of the file, I would suggest to put it at the end or even
strip it completely from the file and keep it somewhere else.

Regards,

Vladimir

4

If you’re on NTFS you could consider keeping it in an alternate stream
in the file.

-p

-----Original Message-----
From: xxxxx@Starbase.com
[mailto:xxxxx@Starbase.com]
Sent: Tuesday, January 14, 2003 10:25 AM
To: NT Developers Interest List

I don’t see any problem with FCB’s FileSize having bigger value than
what you report through other interfaces.
However, I must agreed with David that having header at the beginning of
the file is not the best solution. The only significant advantage I can
see is that you simplify handling of file expansion (only considering
that header has fixed size). However you gain a lot of complications
trying to keep the header out of the cache, dealing with
FileObject->CurrentByteOffset for sequential files (these are just from
the top of my head).
So, unless there are particular reasons why you want to have your header
at the beginning of the file, I would suggest to put it at the end or
even strip it completely from the file and keep it somewhere else.

Regards,

Vladimir

You are currently subscribed to ntdev as: xxxxx@microsoft.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

5

Hi David, Vlad, Peter,

Thanks so much for your helpful suggestions.

  • We keep the fixed-size header data in the file for our own informational purpose, and this header is not necessarily related to the content-mangling algorithm (with which we mangle the actual contents of the file).

  • Even if we don’t keep the header in the beginning of the file, doesn’t the problem of trying to hide it from the Cache Manager still exist?
    (we currently do adjust the FileObject->CurrentByteOffset for certain IRPs to skip the header, and we keep the header in the beginning for only the reasons that Vlad mentioned).

  • Also, let’s assume that we keep no header in the file, our mangling algorithm still increases the length of the file (similar to an encryption algorithm), if the Cache Manager is able to read the “increased size” of the file, but then it is not able to get all the data (because we are un-mangling and giving it the actual data (which is smaller in size)), will that cause problems?

  • The files we mangle can be binaries or data files, e.g. MS Office Programs can certainly use them as documents, or DLLs. (if our driver is loaded, these files will be read correctly, if our driver is not loaded, then the file will be seen as containing garbage data).

  • Lastly, we need to be agnostic of File System types (i.e. we can’t rely on NTFS features, which would have been nice :slight_smile: ).

Thanks again, for taking the time for going through our issues.

-Sincerely,
-Anurag

-----Original Message-----
From: Peter Wieland [mailto:xxxxx@windows.microsoft.com]
Sent: Tuesday, January 14, 2003 10:31 AM
To: NT Developers Interest List
Subject: [ntdev] Re: FS Filter Driver question

If you’re on NTFS you could consider keeping it in an alternate stream
in the file.

-p

-----Original Message-----
From: xxxxx@Starbase.com
[mailto:xxxxx@Starbase.com]
Sent: Tuesday, January 14, 2003 10:25 AM
To: NT Developers Interest List

I don’t see any problem with FCB’s FileSize having bigger value than
what you report through other interfaces.
However, I must agreed with David that having header at the
beginning of
the file is not the best solution. The only significant advantage I can
see is that you simplify handling of file expansion (only considering
that header has fixed size). However you gain a lot of complications
trying to keep the header out of the cache, dealing with
FileObject->CurrentByteOffset for sequential files (these are just from
the top of my head).
So, unless there are particular reasons why you want to have
your header
at the beginning of the file, I would suggest to put it at the end or
even strip it completely from the file and keep it somewhere else.

Regards,

Vladimir

You are currently subscribed to ntdev as: xxxxx@microsoft.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntdev as: xxxxx@greenborder.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

6

The only way I’ve seen this work is to construct a filter that works much
like the compression support for NTFS - that is, your “filter” integrates
into the cache manager and then creates different file objects which it
sends to the underlying FSD. The version YOU maintain in the cache has the
right length/size information, which is what will be used by application
programs.

Then your “filter” calls the underlying FSD to obtain the data (in mangled +
offset) form. That the file size underneath you is different doesn’t
matter.

Of course, when you are done what you have is more like a stacked file
system than a filter - these are the most complex filters that I’ve seen,
and I think are harder to develop than a file system.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Anurag Sharma [mailto:xxxxx@greenborder.com]
Sent: Tuesday, January 14, 2003 4:41 PM
To: NT Developers Interest List
Subject: [ntdev] Re: FS Filter Driver question

Hi David, Vlad, Peter,

Thanks so much for your helpful suggestions.

  • We keep the fixed-size header data in the file for our own informational
    purpose, and this header is not necessarily related to the content-mangling
    algorithm (with which we mangle the actual contents of the file).

  • Even if we don’t keep the header in the beginning of the file, doesn’t the
    problem of trying to hide it from the Cache Manager still exist?
    (we currently do adjust the FileObject->CurrentByteOffset for certain IRPs
    to skip the header, and we keep the header in the beginning for only the
    reasons that Vlad mentioned).

  • Also, let’s assume that we keep no header in the file, our mangling
    algorithm still increases the length of the file (similar to an encryption
    algorithm), if the Cache Manager is able to read the “increased size” of the
    file, but then it is not able to get all the data (because we are
    un-mangling and giving it the actual data (which is smaller in size)), will
    that cause problems?

  • The files we mangle can be binaries or data files, e.g. MS Office Programs
    can certainly use them as documents, or DLLs. (if our driver is loaded,
    these files will be read correctly, if our driver is not loaded, then the
    file will be seen as containing garbage data).

  • Lastly, we need to be agnostic of File System types (i.e. we can’t rely on
    NTFS features, which would have been nice :slight_smile: ).

Thanks again, for taking the time for going through our issues.

-Sincerely,
-Anurag

-----Original Message-----
From: Peter Wieland [mailto:xxxxx@windows.microsoft.com]
Sent: Tuesday, January 14, 2003 10:31 AM
To: NT Developers Interest List
Subject: [ntdev] Re: FS Filter Driver question

If you’re on NTFS you could consider keeping it in an alternate stream
in the file.

-p

-----Original Message-----
From: xxxxx@Starbase.com
[mailto:xxxxx@Starbase.com]
Sent: Tuesday, January 14, 2003 10:25 AM
To: NT Developers Interest List

I don’t see any problem with FCB’s FileSize having bigger value than
what you report through other interfaces.
However, I must agreed with David that having header at the
beginning of
the file is not the best solution. The only significant advantage I can
see is that you simplify handling of file expansion (only considering
that header has fixed size). However you gain a lot of complications
trying to keep the header out of the cache, dealing with
FileObject->CurrentByteOffset for sequential files (these are just from
the top of my head).
So, unless there are particular reasons why you want to have
your header
at the beginning of the file, I would suggest to put it at the end or
even strip it completely from the file and keep it somewhere else.

Regards,

Vladimir

You are currently subscribed to ntdev as: xxxxx@microsoft.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntdev as: xxxxx@greenborder.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntdev as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

7

Ways to keep the header out of the cache are conceptually different for
those two cases. In case if header is at the end you may not even care if
it gets into the cache (unless you don’t want to expose its content). And
event if you don’t want anybody to see what is in it, you can simply get
its valid content in the read completion routine and then fill the buffer
with some garbage. But if you have the header at the beginning you can’t
afford it to get cached at all because in this case you will end up
screwing actual file content when file gets memory mapped. I’m not saying
it is impossible. I just think that avoiding this problem will give you
more headaches than supporting file expansion with the header at the end.

Regards,

Vladimir

8

Tony:

Isn’t “shrinking” bigger file into cache significantly different than
expanding smaller file? Since in this case CM will allocate enough pages
to fit actual file content?
And what does it mean “filter" integrates into the cache manager”? Do you
mean that filter will initialize (and whole 9 yards) FO that it receives
in the create dispatch and use actual FS just to read/write mangled file?
Or there is something else?

TIA,

Vladimir