Role of FO Section Object Pointer

Excuse me if this question has been previously addressed. It relates to the role of a File Object (FO)?s SectionObjectPointer and the use and management of the data to which it points.

First some background: I am writing a minifilter driver that implements copy-on-write. The goal (I?m simplifying this) is to maintain the C: drive unchanged. Any changed files are copied to the (say) D: drive and modified and subsequently served from there. This indirection is transparent to users. For example, if C:\JOE has been modified, an open of C:\JOE actually sees the data on D:\JOE and changes affect it, not C:\JOE.

This is implemented with proxy file objects. The minifilter poses as a file system, responding to file requests, and does so by making separate requests down the filter stack. It might handle requests to open and read C:\JOE by opening and reading file D:\JOE. Or C:\JOE. The point here is that the file object presented to the minifilter (the ?user FO?) is not passed down the stack. Instead a ?proxy file object?, created when the minifilter opens C:\JOE or D:\JOE, is passed down the stack to (ultimately) NTFS. [Sorry if you experts have some term other than ?proxy? for this concept.]

The minifilter has its own implementation for context, stream and file control blocks (CCBs, SCBs and FCBs). It does the right thing during file create/open, hooking a new CCB to the provided file object?s FsContext2, a SCB (created if necessary, one per stream common to all its CCBs) to the FO?s FsContext, and a SECTION_OBJECT_POINTERS struct within the SCB to the FO?s SectionObjectPointer.

Next the problem: When the minifilter handles Ping.exe, it does not execute correctly. (I?m sure many other executables are also problematic.) PING 10.1.2.16 runs just fine, showing reply time etc. PING fails with ?could not find host?.

Next some diagnostic information: I have compared the responses returned by the minifilter for Ping.exe with those return by NTFS when the minifilter is disabled. They are identical. The same Opens, Cleanups, Closes, Reads, Query Infos, and Query Security calls all return the same results. I have extensive tracing to show this and I have run ProcMonitor to confirm it. I have run Ping in user mode under WinDbg both with and without the minifilter and compared the disassembled text. It is identical.

Next the crux: Recall that I said the minifilter sets the FO?s SectionObjectPointer to a (initially null filled) SECTION_OBJECT_POINTER struct within a minifilter SCB. If instead of doing that the minifilter sets the User FO?s SectionObjectPointer to the proxy FO?s SectionObjectPointer (that is, a pointer returned by NTFS) then Ping works just fine. So: UserFO->SOP == proxyFO->SOP => happiness. UserFO->SOP == &SCB->SOP_struct => misery.

Finally the questions: What?s wrong with a minifilter stream having its own SOP struct? It is an abstract stream so why shouldn?t it have its own data section, control section and shared cache map? Shouldn?t Windows simply construct them as needed. Shouldn?t they just work? What ?secret sauce? does the NTFS SOP struct contain that the minifilter SOP doesn?t have?

> Next the problem: When the minifilter handles Ping.exe, it does not execute correctly. (I?m sure

many other executables are also problematic.) PING 10.1.2.16 runs just fine, showing reply time etc.

I cannot understand how PING is related to FS filters.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

The issue is executable image sections, not ping itself. Ping is an
example of one such executable that doesn’t execute correctly while
others do. The point being that the filter in question is causing this
behavior. The question asked is:

“Finally the questions: What?s wrong with a minifilter stream having
its own SOP struct? It is an abstract stream so why shouldn?t it
have its own data section, control section and shared cache map?
Shouldn?t Windows simply construct them as needed. Shouldn?t they
just work? What ?secret sauce? does the NTFS SOP struct contain that
the minifilter SOP doesn?t have?”

Ping is an example used to illustrate the failure.

Mark Roddy

On Thu, Feb 4, 2010 at 2:56 PM, Maxim S. Shatskih
wrote:
>> Next the problem: ?When the minifilter handles Ping.exe, it does not execute correctly. ?(I?m sure
>>many other executables are also problematic.) ?PING 10.1.2.16 runs just fine, showing reply time etc.
>
> I cannot understand how PING is related to FS filters.
>
> –
> Maxim S. Shatskih
> Windows DDK MVP
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

I’ll start by saying that what you’re working with is a layered FSD model,
which is just about as complicated as it gets. You have all the fun and
annoyance of a file system filter with the added benefit of also needing to
create a file system. Fun stuff!

[Sorry if you experts have some term other than ?proxy? for this concept.]

Typically called a shadow file object approach. I’m curious why you chose
this approach in the first place? Based on what you’ve described, it seems
as if you could do this with reparsing (and elbow grease…and possibly
scotch). Unless D:\JOE is just a log or diff of what’s changed in C:\JOE
instead of the most current version of JOE, in which case you have a more
difficult architecture and you’ll likely need the shadow file object
approach (though I still might try architecting myself out of that, it would
depend on the requirements).

Finally the questions: What?s wrong with a minifilter stream having its
own SOP struct?

Nothing architecturally, this is a technique that works (we do it in the
DMK).

Shouldn?t Windows simply construct them as needed. Shouldn?t they just
work? What ?secret sauce? does the NTFS SOP >struct contain that the
minifilter SOP doesn?t have

If you want to split the file objects and have the files independently
cached, then your filter has to interact with the cache manager to initiate
caching on the file, copy to/from the cache, service the paging I/O, etc.
None of this magically happens for you (unfortunately), you have to manage
the upper edge interface in the same manner that a full file system would.

If you dump out the section object pointers structures I suspect that you’ll
see your SOP having a NULL SharedCacheMap and DataSectionObject and the
underlying FSD’s being correctly populated, which would be what the NTFS
structure has that you don’t.

By setting your SOP to the SOP of the underlying FSD you’re bypassing your
filter for some requests and you’re effectively letting the underlying FSD
manage the cache on your file object, which is generally what you’re trying
to avoid in the shadow file object model. The fact that it works is slightly
miraculous, though probably wouldn’t stand up to severe beating.

I suspect that your lack of full cache manager integration is the root of
the problem. For guidance on what that integration looks like check the FAT
source, though I’ll warn that it’s a long road to hoe and you should
probably determine if you really need to go that route first.

In terms of the reported problem, I’d try this on FAT drives with an
instrumented version of FAT so that I could see the difference in requests
received in the D:\ FAT instance with the SOP fudging. The problem with
trying to trace these kinds of problems with the various existing utilities
is that they only show what they show, which can be misleading.

HTH and good luck!

-scott


Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntfsd…
> Excuse me if this question has been previously addressed. It relates to
> the role of a File Object (FO)?s SectionObjectPointer and the use and
> management of the data to which it points.
>
> First some background: I am writing a minifilter driver that implements
> copy-on-write. The goal (I?m simplifying this) is to maintain the C:
> drive unchanged. Any changed files are copied to the (say) D: drive and
> modified and subsequently served from there. This indirection is
> transparent to users. For example, if C:\JOE has been modified, an open
> of C:\JOE actually sees the data on D:\JOE and changes affect it, not
> C:\JOE.
>
> This is implemented with proxy file objects. The minifilter poses as a
> file system, responding to file requests, and does so by making separate
> requests down the filter stack. It might handle requests to open and read
> C:\JOE by opening and reading file D:\JOE. Or C:\JOE. The point here is
> that the file object presented to the minifilter (the ?user FO?) is not
> passed down the stack. Instead a ?proxy file object?, created when the
> minifilter opens C:\JOE or D:\JOE, is passed down the stack to
> (ultimately) NTFS. [Sorry if you experts have some term other than
> ?proxy? for this concept.]
>
> The minifilter has its own implementation for context, stream and file
> control blocks (CCBs, SCBs and FCBs). It does the right thing during file
> create/open, hooking a new CCB to the provided file object?s FsContext2, a
> SCB (created if necessary, one per stream common to all its CCBs) to the
> FO?s FsContext, and a SECTION_OBJECT_POINTERS struct within the SCB to the
> FO?s SectionObjectPointer.
>
> Next the problem: When the minifilter handles Ping.exe, it does not
> execute correctly. (I?m sure many other executables are also
> problematic.) PING 10.1.2.16 runs just fine, showing reply time etc.
> PING fails with ?could not find host?.
>
> Next some diagnostic information: I have compared the responses returned
> by the minifilter for Ping.exe with those return by NTFS when the
> minifilter is disabled. They are identical. The same Opens, Cleanups,
> Closes, Reads, Query Infos, and Query Security calls all return the same
> results. I have extensive tracing to show this and I have run ProcMonitor
> to confirm it. I have run Ping in user mode under WinDbg both with and
> without the minifilter and compared the disassembled text. It is
> identical.
>
> Next the crux: Recall that I said the minifilter sets the FO?s
> SectionObjectPointer to a (initially null filled) SECTION_OBJECT_POINTER
> struct within a minifilter SCB. If instead of doing that the minifilter
> sets the User FO?s SectionObjectPointer to the proxy FO?s
> SectionObjectPointer (that is, a pointer returned by NTFS) then Ping works
> just fine. So: UserFO->SOP == proxyFO->SOP => happiness. UserFO->SOP ==
> &SCB->SOP_struct => misery.
>
> Finally the questions: What?s wrong with a minifilter stream having its
> own SOP struct? It is an abstract stream so why shouldn?t it have its
> own data section, control section and shared cache map? Shouldn?t Windows
> simply construct them as needed. Shouldn?t they just work? What ?secret
> sauce? does the NTFS SOP struct contain that the minifilter SOP doesn?t
> have?
>
>

The minifilter intercepts file requests for all files on the C: drive.
If one runs Ping from user mode then Windows opens c:\windows\system32\ping.exe, performs various query info functions, does paging reads, cleanup, close etc.
The minifilter handles all these file requests.

Ping is just an example of an executable file that does not run correctly notwithstanding that all file operations seem to be satisfied correctly.

We’ve actually seen issues in which some applications do not behave properly when a file is redirected to a different location; in the case of network applications this usually is because the filter manager cannot get back the name expected by the application (e.g., the firewall software.) It might be a bit of a surprise to folks, but it now turns out that non-FS filters routinely use the filter manager name query operations to retrieve file names.

The manifestations I recall is how https operations would mysteriously fail - and it tracked back to the name being retrieved by the filter manager.

Another interesting case (and one that I debugged, so I still recall the pain) is that win32k.sys would open up the regionalized keyboard DLL and then query the name - if the name didn’t come back the same, it would choke. This SPECIFICALLY caused problems in the redirection case because the “device name” bit would come back with the name of the (redirected) device which didn’t match the original device (so the original device was “\Device\Mumble\Windows\System32\kbdus.dll” and the name it got back was “\Device\AlternateMumble\Windows\System32\kbdus.dll” and the machine would refuse to boot.)

Tony
OSR

> This is implemented with proxy file objects. The minifilter poses as a file system, responding to file

requests, and does so by making separate requests down the filter stack.

I’m not sure the layered FSD (and this is what you’re speaking about, when app’s file objects are not visible to the real FSD and only to your filter) is implementable in FltMgr’s minifilter.

With a layered FSD, you need to create your own FCB (FCB header with file sizes and locks and SOP), set ->FsContext and ->SectionObjectPointer to this structure of your own, reference-count the FCB, and also do all of the Cc/Mm interaction which the real FSD like FASTFAT does.

In this case you’re writing an FSD, not a filter.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

>I’m not sure the layered FSD (and this is what you’re speaking about, when

app’s file objects are not visible to the real FSD and >only to your
filter) is implementable in FltMgr’s minifilter.

For a while we had two implementations in the DMK, one legacy and one
minifilter. We’ve finally just ditched the legacy version, which makes
everyone around here very happy (I was getting sick of hacking around Win2K
bugs)…

-scott


Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntfsd…
>> This is implemented with proxy file objects. The minifilter poses as a
>> file system, responding to file
>>requests, and does so by making separate requests down the filter stack.
>
> I’m not sure the layered FSD (and this is what you’re speaking about, when
> app’s file objects are not visible to the real FSD and only to your
> filter) is implementable in FltMgr’s minifilter.
>
> With a layered FSD, you need to create your own FCB (FCB header with file
> sizes and locks and SOP), set ->FsContext and ->SectionObjectPointer to
> this structure of your own, reference-count the FCB, and also do all of
> the Cc/Mm interaction which the real FSD like FASTFAT does.
>
> In this case you’re writing an FSD, not a filter.
>
> –
> Maxim S. Shatskih
> Windows DDK MVP
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>

> For a while we had two implementations in the DMK, one legacy and one

minifilter.

Thanks, good to know this.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

The shadow file object implementation where you use the SOP of the FO below obviously only works if you don’t need to perform any modification on the data (i.e. the data at your level is the same as the data the FS level). We’ve tried this internally (obviously in a minifilter) and it seems to work but we’re still debating if this is architecturally a good idea (we haven’t found any issues so far though). It is not useful for filters that need to transform the data.

I think the issue could be what Tony mentioned, which is that some component in the OS (like a firewall) might get a wrong name for the application and not let it query the IP for the server name call ping with.

I believe that there is one thing that Procmon doesn’t show, which is name requests (at least, I think it doesn’t). I would suggest you implement a simple pass-through name provider (for testing purposes only) and have it DbgPrint the names it sees in both cases. This is where I expect you’ll see a difference.

Does your minifilter implement name provider callbacks ?

Regards,
Alex.
This posting is provided “AS IS” with no warranties, and confers no rights.

The answer proved to be implementing the name provider callbacks, as suggested by Alex and Tony.
Although documented as “optional”, it appears that they are in fact essential.
Thanks to all who responded.

Hello Robert,

I’m glad that you managed to fix the problem. As to what optional really means, I guess in this case it means that you can write some classes of minifilters without implementing them :). I find more and more issues where the root problem is due to the inherent vagueness of human languages (when talking about things like “optional” or “synchronous” or “virtualized” and everybody gets a different meaning).

I would also like to point out that filter manager driver verifier (i.e. enabling driver verifier for both filter manager and your minifilter) is very powerful and has been improved a lot for Win7, so even if you don’t plan to ship on Win7 (yet :slight_smile: ) testing on it with driver verifier might find interesting problems. I’m bringing this up because it would detect this problem (not implementing name providers in a context where they are actually not optional).

I wrote a small blog post on it, here: http://blogs.msdn.com/alexcarp/archive/2009/10/08/verifier-checks-a-filter-has-completed-an-operation-but-does-not-implement-name-provider-callbacks.aspx

Thanks,
Alex.

Thanks, Alex, for the additional advice and link-to-blog.

The documentation of PFLT_GENERATE_FILE_NAME() [ms793795.aspx] is … sparse.
What should the callback return if it is passed an unknown FileObject (for example one that the minifilter chose not to handle during IRP_MJ_CREATE but instead passed down the filter stack)?

There no obvious way for the minifilter to say “Not one of mine; ask another filter lower down to generate the name!”

I tried returning STATUS_INVALID_HANDLE. That results in the misery that started this thread.

Simply query the file name using FltGetFileNameInformation and return the
results in this case. It is *kinda* the same thing as passing it down.
It’s like doing FltReadFile in a pre-read callback instead of sending the
original read down the stack.
A name generation callback MUST handle all file objects/ callback data for
which name has been requested.

Regards,
Ayush Gupta
AI Consulting

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@virtualcomputer.com
Sent: Friday, February 12, 2010 7:53 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Role of FO Section Object Pointer

Thanks, Alex, for the additional advice and link-to-blog.

The documentation of PFLT_GENERATE_FILE_NAME() [ms793795.aspx] is …
sparse.
What should the callback return if it is passed an unknown FileObject (for
example one that the minifilter chose not to handle during IRP_MJ_CREATE but
instead passed down the filter stack)?

There no obvious way for the minifilter to say “Not one of mine; ask another
filter lower down to generate the name!”

I tried returning STATUS_INVALID_HANDLE. That results in the misery that
started this thread.


NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer