questions about the stream file object

Hi,Dear all :slight_smile:

I have some questions about the stream file object,hope somebody can help me…

  1. In what situation that system will create a stream file object? Does it mean when user application use C++'s iostream operation will cause this happen?
  2. what’s the different between a stream file object and a normal file object?
  3. MSDN mention that when a stream file object created, It does not send a IRP_MJ_CREATE IRP but a IRP_MJ_CLEANUP IRP,in this case,how do we catch this file create operation in our filter?

xxxxx@21cn.com wrote:

Hi,Dear all :slight_smile:

I have some questions about the stream file object,hope somebody can help me…

  1. In what situation that system will create a stream file object? Does it mean when user application use C++'s iostream operation will cause this happen?

Other kernel mode components can use stream file objects in a wide range
of designs. NTFS uses SFO’s for cache map initialization, hence you see
paging coming in against a SFO.

  1. what’s the different between a stream file object and a normal file object?

The difference is that you have not seen an IRP_MJ_CREATE for the file
object and the FO_STREAM_FILE_OBJECT Flag is set. Other than that they
can be used for any operation that comes through the file system stack.

  1. MSDN mention that when a stream file object created, It does not send a IRP_MJ_CREATE IRP but a IRP_MJ_CLEANUP IRP,in this case,how do we catch this file create operation in our filter?

You can not capture the creation of these through standard mechanisms.
You do need to recognize the tear down of these during IRP_MJ_CLEANUP
and CLOSE processing though. Have you read the reference counting
article at OSR Online? This discusses how to make sure your counts are
correct. Of course, with Filter Manager you don’t need to worry about
them from the perspective of reference counting.

Pete


NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295

Pete,thanks for your reply: )
I am a new guy in filter developing,can you give me the link about the reference counting article at OSR Online?

I imagine that this is it:

http://www.osronline.com/article.cfm?id=102

xxxxx@21cn.com wrote:

Pete,thanks for your reply: )
I am a new guy in filter developing,can you give me the link about the reference counting article at OSR Online?

by the way Pete,can you give me links about stream file object, I should learn more about it, thanks

xxxxx@21cn.com wrote:

by the way Pete,can you give me links about stream file object, I should learn more about it, thanks

I would read up on the WDK docs about the IoCreateStreamFileObject(),
IoCreateStreamFileObjectEx() and IoCreateStreamFileObjectLite(). They
each offer slightly different functionality but do give information on
what they can be used for. Other than that, search for the APIs in the
FAT source code and follow how they are used.

Pete


NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295

copy that~:)

> 1. In what situation that system will create a stream file object? Does it mean when user application

use C++'s iostream operation will cause this happen?

No for sure.

C++ streams are the wrappers around the CRT’s read/write/open/creat/close, and they are - in Windows - in turn wrappers over kernel32!CreateFile/ReadFile, which are in turn wrappers over ntdll!NtCreateFile/NtReadFile syscalls going down to the kernel.

Stream file object is the file object handicrafted by the FSD itself without going thru the usual kernel’s IoCreateFile path.

The purpose of such file objects are at FSD’s developer will, but usually they are used to create the cache maps and thus cache some metadata. For instance, FASTFAT uses such object to cache FAT and the FAT12/16 root dir.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

>

  1. In what situation that system will create a stream file object? Does it
    mean when user application use C++'s iostream operation will cause this
    happen?

You may not know when the FileSystem decides to spawn a stream file object.
The scenarios can be many. The most obvious case is when file mapping occurs
and write behind is required.

Imagine the following scenario.

H1 = CreateFile
H2 = CreateFileMapping
Mem = MapviewOfFile
CloseHandle(H1);

Alter Mem -> file get modified, even though to actual HANDLE exists

Close(H2)

This is when the system spawns a stream file object, you may catch creation
in IRP_MJ_ACQUIRE_FOR_SECTION_SYNCH, anyway, about there.
I do not actually recommend using that FO for any operation.

With respect,
Gabriel Bercea

GaMiTech Software Development
Mobile contact: (+40)0740049634
eMail: xxxxx@gmail.com
Blog: http://gamitech.blogspot.com/

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@21cn.com
Sent: Thursday, April 09, 2009 5:44 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] questions about the stream file object

Hi,Dear all :slight_smile:

I have some questions about the stream file object,hope somebody can help
me…

  1. In what situation that system will create a stream file object? Does it
    mean when user application use C++'s iostream operation will cause this
    happen?
  2. what’s the different between a stream file object and a normal file
    object?
  3. MSDN mention that when a stream file object created, It does not send a
    IRP_MJ_CREATE IRP but a IRP_MJ_CLEANUP IRP,in this case,how do we catch this
    file create operation in our filter?

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Hi Gabriel:)
In this case,we don’t know when a stream file object would be created,how to let the filter recognize the FO is a stream file object or not? I find a doc in OSR,it mention a way to deal with the FO,the original text is :

"Managing File Objects

Few things continue to confound file system filter driver writers more than the issue of how to handle file objects within their filter driver. In Windows 2000, the introduction of a new function, IoCreateStreamFileObjectLite has made their task even more difficult than it was in the past.

Just to remind all of the file system filter driver writers: File Objects are not files. They are references to the actual file. Each time the file is opened, a new file object is created. Applications can open files. Filter drivers can open files. File Systems can open files. Many files are opened via IoCreateFile (this is the path that is used by applications and many kernel mode components). A few files are opened using IoCreateStreamFileObject and IoCreateStreamFileObject Lite.

Of course, a typical file system filter driver associates file objects together by using the FsContext field. While this works with all file systems, we note that in the case of network file systems the values of this field are often updated throughout the life of the file. This is because a network file system may not actually ?open? the file until an application begins to perform I/O against the file. We have even observed some redirectors that leave this field as NULL after the IRP_MJ_CREATE, only filling it in later, when they have the necessary information.

Probably the greatest difficulty facing file system filter driver writers are the file objects created by the file system itself. That is because normally the creation of a new file object is indicated to the filter driver via an IRP_MJ_CREATE but for file objects created by the file systems directly, there is no such indication.

In Windows NT 4.0, filter driver developers found that they did receive an IRP_MJ_CLEANUP for these file objects (created using IoCreateStreamFileObject) but in Windows 2000, the new call IoCreateStreamFileObjectLite eliminates even the IRP_MJ_CLEANUP call.

To work around this, we normally suggest that file system filter driver writers organize their data structures so that they maintain a per-file structure keyed on the FsContext value. Within each of the per-file structures, we then maintain a list of each file object associated with that file.

Then, during the processing of an IRP_MJ_READ, IRP_MJ_WRITE, IRP_MJ_QUERY_INFORMATION, IRP_MJ_SET_INFORMATION, IRP_MJ_CREATE or IRP_MJ_CLEANUP operation we check to see if this file object is currently listed as one of the file objects associated with the given file. If it is not associated with the given file structure, we then associate it. This ensures that we will not discard the per-file structure prematurely.

Some of these files are internal control files for the FSD itself, and in that case your filter may not be tracking a per-file data structure for them at all. In most cases we ignore those files, but if they are important to your specific filter driver, this mechanism will allow you to detect and track them as well.

For network redirectors, we suggest monitoring the FsContext field before, and after, an I/O operation is processed by the underlying FSD; then, if the value of this field changes across the call, the filter can update its data structures accordingly."

Is it a best way to handle normal FO and stream FO?

“Is it a best way to handle normal FO and stream FO?”

Ah, but it all depends on how you “handle” normal FOs. In other words, what do you want to do ? If you want to capture creates… well… you can’t.

Regards,
Alex.
This posting is provided “AS IS” with no warranties, and confers no rights.

> Imagine the following scenario.

H1 = CreateFile
H2 = CreateFileMapping
Mem = MapviewOfFile
CloseHandle(H1);

Alter Mem -> file get modified, even though to actual HANDLE exists

Close(H2)

This is when the system spawns a stream file object

This is the implementation detail of a particular FSD. There is no serious need in this at all.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

Alex,do you have an idea to capture the creates?

Yes, I was just giving an example about how not transparent these operations
can be

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Saturday, April 11, 2009 2:57 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] questions about the stream file object

Imagine the following scenario.

H1 = CreateFile
H2 = CreateFileMapping
Mem = MapviewOfFile
CloseHandle(H1);

Alter Mem -> file get modified, even though to actual HANDLE exists

Close(H2)

This is when the system spawns a stream file object

This is the implementation detail of a particular FSD. There is no serious
need in this at all.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com


NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer