How to detect stream file object creation ?

Hi , guys.

I am working on a FS miniflter driver, I got a problem:
Because FS create stream FO as a side effect of operations other than IRP_MJ_CREATE, how to detect stream FO creation? What should I do if I want to get the original FO?

Thanks in advance.

Alex.

You can check if FO_STREAM_FILE flag is set in fileobject->Flags.
 
Regards,
Ayush Gupta

Bollywood news, movie reviews, film trailers and more! Go to http://in.movies.yahoo.com/

Hi Ayush!

I’m sorry my text my not convey the problem properly.

Look at the prototype:
PFILE_OBJECT
IoCreateStreamFileObject(
IN PFILE_OBJECT FileObject OPTIONAL,
IN PDEVICE_OBJECT DeviceObject OPTIONAL
);

Given a FO(F1) as the first param, this function creates a new FO(F2). Now my problem is:

I have F2, and, I want to get F1.
How can I do this?

Thanks.

Alex.

I think you missed something in that API:

PFILE_OBJECT
IoCreateStreamFileObject(
IN PFILE_OBJECT FileObject OPTIONAL,
IN PDEVICE_OBJECT DeviceObject OPTIONAL
);

Notice that keyword OPTIONAL? That’s intended to tell you it need not
be passed.

In fact, one of the two must be passed because they are used as a
template for building the new file object. But neither is referenced
from the newly created file object.

For *this* particular API, however, you can detect the creation: watch
for the IRP_MJ_CLEANUP call that will occur before it completes. Of
course, the file systems folks at Microsoft caught onto this trick and
added IoCreateStreamFileObjectLite to combat our wily ability to find
their file object - it eliminates that IRP_MJ_CLEANUP.

Worse yet, they might not even USE that new file object until after the
original file object has been torn down (NTFS actually does this.) Yup,
that’s even worse than the case in which the file object wasn’t even
passed.

So the answer to your question (“how do I get the original file object
passed in here”) is “you cannot do this in any reliable fashion.” I
would consider stack backtracking to find it unreliable and aside from
that you’re stuck.

Don’t worry, it’ll get worse from here. You’ve only just begun to feel
the pain.

Tony
OSR

Thanks a lot Tonny, for the explanation.

I will find another way to fix my problem.
Thansk!

Alex.

2008/12/22 Tony Mason

> I think you missed something in that API:
>
> PFILE_OBJECT
> IoCreateStreamFileObject(
> IN PFILE_OBJECT FileObject OPTIONAL,
> IN PDEVICE_OBJECT DeviceObject OPTIONAL
> );
>
> Notice that keyword OPTIONAL? That’s intended to tell you it need not
> be passed.
>
> In fact, one of the two must be passed because they are used as a
> template for building the new file object. But neither is referenced
> from the newly created file object.
>
> For this particular API, however, you can detect the creation: watch
> for the IRP_MJ_CLEANUP call that will occur before it completes. Of
> course, the file systems folks at Microsoft caught onto this trick and
> added IoCreateStreamFileObjectLite to combat our wily ability to find
> their file object - it eliminates that IRP_MJ_CLEANUP.
>
> Worse yet, they might not even USE that new file object until after the
> original file object has been torn down (NTFS actually does this.) Yup,
> that’s even worse than the case in which the file object wasn’t even
> passed.
>
> So the answer to your question (“how do I get the original file object
> passed in here”) is “you cannot do this in any reliable fashion.” I
> would consider stack backtracking to find it unreliable and aside from
> that you’re stuck.
>
> Don’t worry, it’ll get worse from here. You’ve only just begun to feel
> the pain.
>
> Tony
> OSR
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

I am curious as to why a filter would care about this co-relation?

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights

Hi, Sarashh

I save some impotant infomation in stream_handle_context which is created in
my IRP_MJ_CREATE handler. So when I get a stream_file, I cannot find the
infomation I want.

Alex.

2008/12/23

> I am curious as to why a filter would care about this co-relation?
>
> Regards,
> Sarosh.
> File System Filter Lead
> Microsoft Corp
>
> This posting is provided “AS IS” with no warranties, and confers no Rights
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> You are currently subscribed to ntfsd as: xxxxx@gmail.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

Sarosh, this kind of detection is important for applications tracking file
mapping operations, and initial file handle closing before modifying the
memory.

An application doing the following sequence of events:

CreateFile -> Handle1
CreateFileMapping -> Handle2
CloseHandle (Handle1) -> will trigger both cleanup and close
MapViewOfFile (Handle2)
Modify mapped memory
Flush to file (pbably a stream FO)
Unmap
CloseHandle2 ,

will most certainly modify the file, and the file system filter monitoring
the IRP_MJ_CREATE routine will not be able to catch this, because the FS
will spawn a stream file object to flush dirty memory to the file. I’ve seen
these kind of scenarios many time with filespy.
How should these scenarios be approached.

Thanks.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@microsoft.com
Sent: Tuesday, December 23, 2008 3:25 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] How to detect stream file object creation ?

I am curious as to why a filter would care about this co-relation?

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@gmail.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

The purpose of filter contexts is to track the lifetime of the object in
question. Because it is associated with the stream
(FileObject->FsContext, which FAT will use for a “File Control Block”
and NTFS will use for a “Stream Control Block”) it’s lifetime is the
lifetime of that control structure.

Since Filter Manager relies upon filter contexts to track the
mini-filter contexts, you have the same semantics in a mini-filter
driver. If you are still building a legacy filter you can use the same
mechanism (filter contexts.) It is only when you look at situations
prior to filter contexts that require you understand the type of
relationship you describe (I’ve done that and we’ve written several
articles about the “reference counting problem” in The NT Insider.)

Thus, in your scenario, you can associate a filter context with the
initial open:

CreateFile -> Handle1
CreateFileMapping -> Handle2 mapping is done>
CloseHandle (Handle1) -> will trigger both cleanup and close because underlying structure has not been deleted>
MapViewOfFile (Handle2)
Modify mapped memory
Flush to file (pbably a stream FO)
Unmap
CloseHandle2 ,

IRP_MJ_CLOSE on stream FO
structure is deleted>

Tony
OSR

Thanks Tony.
This is the exact mechanism I use I my file system filter (stream context
technique).
I was wondering, could you point me to the links to your written articles.
Thank you and happy holidays.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Tuesday, December 23, 2008 4:58 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] How to detect stream file object creation ?

The purpose of filter contexts is to track the lifetime of the object in
question. Because it is associated with the stream
(FileObject->FsContext, which FAT will use for a “File Control Block”
and NTFS will use for a “Stream Control Block”) it’s lifetime is the
lifetime of that control structure.

Since Filter Manager relies upon filter contexts to track the
mini-filter contexts, you have the same semantics in a mini-filter
driver. If you are still building a legacy filter you can use the same
mechanism (filter contexts.) It is only when you look at situations
prior to filter contexts that require you understand the type of
relationship you describe (I’ve done that and we’ve written several
articles about the “reference counting problem” in The NT Insider.)

Thus, in your scenario, you can associate a filter context with the
initial open:

CreateFile -> Handle1
CreateFileMapping -> Handle2 mapping is done>
CloseHandle (Handle1) -> will trigger both cleanup and close because underlying structure has not been deleted>
MapViewOfFile (Handle2)
Modify mapped memory
Flush to file (pbably a stream FO)
Unmap
CloseHandle2 ,

IRP_MJ_CLOSE on stream FO
structure is deleted>

Tony
OSR


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Slight correction:

CloseHandle (Handle1) -> will trigger both cleanup and close

This should only trigger a cleanup. The close will not come down since
Memory manager will have an outstanding reference to the file object
that backs the mapping.

Also, as Tony, correctly points out you can use filter contexts to track
these IO on the same stream from different file objects. You can attach
contexts at the stream handle (file object), stream or file level, as
appropriate.

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights

Bercea Gabriel wrote:

Sarosh, this kind of detection is important for applications tracking file
mapping operations, and initial file handle closing before modifying the
memory.

An application doing the following sequence of events:

CreateFile -> Handle1
CreateFileMapping -> Handle2
CloseHandle (Handle1) -> will trigger both cleanup and close
MapViewOfFile (Handle2)
Modify mapped memory
Flush to file (pbably a stream FO)
Unmap
CloseHandle2 ,

will most certainly modify the file, and the file system filter monitoring
the IRP_MJ_CREATE routine will not be able to catch this, because the FS
will spawn a stream file object to flush dirty memory to the file. I’ve seen
these kind of scenarios many time with filespy.
How should these scenarios be approached.

Thanks.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@microsoft.com
Sent: Tuesday, December 23, 2008 3:25 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] How to detect stream file object creation ?

I am curious as to why a filter would care about this co-relation?

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@gmail.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Hi Sarosh

This might not be 100% “on message” but, anyway, here we go :slight_smile:

Consider FSCTL_SET_COMPRESSION for some file object FO1. So as this is processed in NTFS, file system does MdlRead/MdlWrite for some file object FO2 which happens to be a stream file object, related to FO1. This happens to be part of the FSCTL_SET_COMPRESSION processing in NTFS.

Let’s imagine a (mini) filter which will see these things, and for whatever reason is required to (in some manner, not important here now) handle these MdlRead/MdlWrite different from other instances of the same operation, because these are uninteresting features of the processing of FSCTL_SET_COMPRESSION … but the “some manner” depends on the name used to open FO1. Here, regards “the name used to open” we should not overlook NTFS hard links.

If you like, I can provide less “theoretical” details, but off-list. Please let me know.

Best Wishes
Lyndon

I probably observed stream FOs while working with MM-files mapped to
some user process. The scenario was the following:

  1. FltCreateFileEx() -> returned FileHandle & FileObject (FileObject
    with HandleCount == 1 & PointerCount == 2)
  2. ZwCreateSection() -> returned SectionHandle (SectionObject with
    HandleCount == 1 & PointerCount == 1, FileObject as before with
    HandleCount == 1 & PointerCount == 2, and SectionObject is already
    associated with quite a different FileObject)
  3. ZwMapViewOfSection() -> returned some buffer pointer (SectionObject
    as before with HandleCount == 1 & PointerCount == 1)
  4. ObDereferenceObject() & FltClose() for FileObject and FileHandle
    respectively (so for now, as I understand, FileObject is completely
    ready for teardown since it only had 2 references)
  5. ZwClose() for SectionHandle (so SectionObject is also totally ready
    for teardown since it had 1 reference)

So what we got at the end? We got some FileObject with HandleCount == 0
& PointerCount == 1 which backs up the created memory section. But what
about the original FileObject (why not to use it for consequent paging
I/O?) and also SectionObject which had only 1 reference before
ZwClose()? Earlier I think that while working with MM-files a FileObject
is referenced by a corresponding SectionObject and the latter is
referenced by mapped views. And will mapped pages (and corresponding
section object) be released when a process which this section is mapped
to is terminated?

Sorry for some kind of off-topic message, but I think there’s also a
matter of stream FOs in this case.

WBR,
Konstantin Manurin

Bercea Gabriel wrote:

Sarosh, this kind of detection is important for applications tracking file
mapping operations, and initial file handle closing before modifying the
memory.

An application doing the following sequence of events:

CreateFile -> Handle1
CreateFileMapping -> Handle2
CloseHandle (Handle1) -> will trigger both cleanup and close
MapViewOfFile (Handle2)
Modify mapped memory
Flush to file (pbably a stream FO)
Unmap
CloseHandle2 ,

will most certainly modify the file, and the file system filter monitoring
the IRP_MJ_CREATE routine will not be able to catch this, because the FS
will spawn a stream file object to flush dirty memory to the file. I’ve seen
these kind of scenarios many time with filespy.
How should these scenarios be approached.

Thanks.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@microsoft.com
Sent: Tuesday, December 23, 2008 3:25 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] How to detect stream file object creation ?

I am curious as to why a filter would care about this co-relation?

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@gmail.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

ZwCreateSection should have bumped up the reference on the FileObject in
(2) if MM was planning to use that file object for paging IO. However,
MM probably already had a FileObject referenced for that stream. Hence
you never saw a ref count increase on the file object that you passed in
to ZwCreateSection.

If you put a stream context on the file object on which ZwCreateSection
was called, it will also be a stream context for the other file object
that MM has. So when you see paging IO and query for a context, you will
see the stream context that you set on the file object from ZwCreateSection.

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights

Konstantin Manurin wrote:

I probably observed stream FOs while working with MM-files mapped to
some user process. The scenario was the following:

  1. FltCreateFileEx() -> returned FileHandle & FileObject (FileObject
    with HandleCount == 1 & PointerCount == 2)
  2. ZwCreateSection() -> returned SectionHandle (SectionObject with
    HandleCount == 1 & PointerCount == 1, FileObject as before with
    HandleCount == 1 & PointerCount == 2, and SectionObject is already
    associated with quite a different FileObject)
  3. ZwMapViewOfSection() -> returned some buffer pointer (SectionObject
    as before with HandleCount == 1 & PointerCount == 1)
  4. ObDereferenceObject() & FltClose() for FileObject and FileHandle
    respectively (so for now, as I understand, FileObject is completely
    ready for teardown since it only had 2 references)
  5. ZwClose() for SectionHandle (so SectionObject is also totally ready
    for teardown since it had 1 reference)

So what we got at the end? We got some FileObject with HandleCount == 0
& PointerCount == 1 which backs up the created memory section. But what
about the original FileObject (why not to use it for consequent paging
I/O?) and also SectionObject which had only 1 reference before
ZwClose()? Earlier I think that while working with MM-files a FileObject
is referenced by a corresponding SectionObject and the latter is
referenced by mapped views. And will mapped pages (and corresponding
section object) be released when a process which this section is mapped
to is terminated?

Sorry for some kind of off-topic message, but I think there’s also a
matter of stream FOs in this case.

WBR,
Konstantin Manurin

Bercea Gabriel wrote:
> Sarosh, this kind of detection is important for applications tracking file
> mapping operations, and initial file handle closing before modifying the
> memory.
>
> An application doing the following sequence of events:
>
> CreateFile -> Handle1
> CreateFileMapping -> Handle2
> CloseHandle (Handle1) -> will trigger both cleanup and close
> MapViewOfFile (Handle2)
> Modify mapped memory
> Flush to file (pbably a stream FO)
> Unmap
> CloseHandle2 ,
>
> will most certainly modify the file, and the file system filter monitoring
> the IRP_MJ_CREATE routine will not be able to catch this, because the FS
> will spawn a stream file object to flush dirty memory to the file. I’ve seen
> these kind of scenarios many time with filespy.
> How should these scenarios be approached.
>
> Thanks.
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of
> xxxxx@microsoft.com
> Sent: Tuesday, December 23, 2008 3:25 AM
> To: Windows File Systems Devs Interest List
> Subject: RE:[ntfsd] How to detect stream file object creation ?
>
> I am curious as to why a filter would care about this co-relation?
>
> Regards,
> Sarosh.
> File System Filter Lead
> Microsoft Corp
>
> This posting is provided “AS IS” with no warranties, and confers no Rights
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> You are currently subscribed to ntfsd as: xxxxx@gmail.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
>

I guess the point I am trying to make is that the file system may use a
stream file object to represent and process a stream internally. This
operation is internal to the file system and trying to co-relate this to
some operation on a user opened file object, seems to be an attempt to
peek into the internal workings of the file system. Such dependencies
are pretty likely to be broken from release to release…

If you have a good scenario where you need this feel to send me mail
offlist or we can chat at the upcoming plugfest.

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights

xxxxx@neverfailgroup.com wrote:

Hi Sarosh

This might not be 100% “on message” but, anyway, here we go :slight_smile:

Consider FSCTL_SET_COMPRESSION for some file object FO1. So as this is processed in NTFS, file system does MdlRead/MdlWrite for some file object FO2 which happens to be a stream file object, related to FO1. This happens to be part of the FSCTL_SET_COMPRESSION processing in NTFS.

Let’s imagine a (mini) filter which will see these things, and for whatever reason is required to (in some manner, not important here now) handle these MdlRead/MdlWrite different from other instances of the same operation, because these are uninteresting features of the processing of FSCTL_SET_COMPRESSION … but the “some manner” depends on the name used to open FO1. Here, regards “the name used to open” we should not overlook NTFS hard links.

If you like, I can provide less “theoretical” details, but off-list. Please let me know.

Best Wishes
Lyndon

> about the original FileObject (why not to use it for consequent paging

I/O?) and also SectionObject which had only 1 reference before
ZwClose()?

SectionObject is a tiny structure which has a) page protection flags b) reference to the “data control area” (MmCa).

MmCa is the real structure which owns the physical pages of the file. There can be only 0 or 1 existing data control areas per FCB (but there can be many section objects - each ZwCreateSection creates one). One of the pointers in SECTION_OBJECT_POINTERS in the FCB points at MmCa (and can be NULL).

MmCa is not an Ob’s object and cannot have handles to it, but it holds a ref on the file object it was created for, the file object in turn holds a ref to FCB (maintained by FSD), and FCB’s SECTION_OBJECT_POINTERS holds a weak ref back to MmCa.

So:

  • CreateFile create a file object and a handle to it (File 0 ref 1 handle).
  • CreateFileMapping creates MmCa and a section object (Sect 0 ref 1 handle, MmCa 1 ref, File 1 ref 1 handle - 1 ref from MmCa).
  • CloseHandle of the file handle causes MJ_CLEANUP, after this: File 1 ref (from MmCa) 0 handle.
  • MapViewOfFile creates a VAD which references the MmCa (MmCa 2 refs, File 1 ref 0 handle).
  • CloseHandle for the section destroys Sect and derefs MmCa once (MmCa 1 ref from mapped VAD, File 1 ref from MmCa 0 handle).

After this, all paging IO initiated by MM goes thru the file object referenced by MmCa, i.e. original file object created in CreateFile.

As about the physical pages associated with the file data - they are actually the innerworkings of MmCa.

Now let’s add Cc to picture. Cc’s shared cache map holds a ref to MmCa. As about the life time of CcSc itself - it is governed a) by lack of CcUninitializeCacheMap, before this CcSc cannot die, and this is called in MJ_CLEANUP b) by the garbage collection inside Cc which will allow the idle CcSc+MmCa to live for some time (and to survive quick reopens of the same file - next CcInitializeCacheMap will just pick existing CcSc from the GC list and resurrect it), and be garbage-collected later.

So, until Cc will do the garbage collection, CcSc/MmCa pair is alive and holds a ref on the file object. That’s why MJ_CLOSE can be delayed a lot after MJ_CLEANUP.

I don’t know how stream file objects go to this picture. I think they are only used to register the metadata streams with Cc - MFT, bitmaps, FATs and so on. More so, even directories must have a stream file object, since the directory file object exposed to user mode (result of ZwCreateFile for a directory) is just an enumeration context for ZwQueryDirectoryFile and is hardly the file object suitable for Cc.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

Actually Max, the challenging case is NTFS, where it often chooses to
use a stream file object (of its own creation) to pass into
CcInitializeCacheMap. I’m not sure of the reason for this, but I have
observed this numerous times over the years (as with anything when
peeking inside the behavior, your mileage may vary. Those darned NTFS
developers keep making small tweaks to the code.)

Thus, it is that stream file object that ends up backing the cache.

Of course, the FAT example code only shows them being used for meta-data
structures; NTFS however uses them extensively internally. We’ve done
similar things with our own work in our data modification kit (in fact,
I JUST ran into this rather interesting bug in one of the Kaspersky A/V
product in which it sends a file object that points to OUR FSD to NTFS.
I suspect I know why, but of course NTFS just doesn’t like it when you
send another FSDs file objects to it.)

This should all “just work” but relies upon people knowing and following
the rules (like “when given a file object, I use FO->Vpb->DO for
passing, not FO->DO->Vpb->DO,” since the former will point to the file
system that owns the FO, and the latter points to the file system that
owns the volume. Usually they are the same, but when they aren’t, bad
things happen.)

Naturally, the very reason this list exists is because those rules
aren’t published and they certainly aren’t intuitively obvious to even
the most experienced observer.

Tony
OSR

Hi Bercea,

See http://www.osronline.com/article.cfm?id=102. There are other
tangential (albeit interesting) articles about reference counting in
other situations as well - just search for “reference count” and you’ll
see both NTFSD discussions as well as those other articles.

Is it the holidays already? Hmm, guess that explains why it’s been so
quiet around the OSR office. I suppose that means I should go get ready
for them, no?

Tony
OSR

Thank you Tony for the tip (article).

Is it the holidays already? Hmm, guess that explains why it’s been so quiet around the OSR office.

Yep, it’s true, better start celebrating :slight_smile: