Missing IRP_MJ_CREATE

NTFSD
1

I’m tracking filenames in a filter driver, hooking into IRP_MJ_CREATE. This is not always called and I’m missing the creation of a bunch of files. Looks to me like this may be caused by IoCreateStreamFileObject bypassing the IRP_MJ_CREATE. Anyone got any suggestions on how to get the missing filename data?

2

>I’m tracking filenames in a filter driver, hooking into IRP_MJ_CREATE.

This is not always called and I’m missing the creation >of a bunch of
files.

You’re missing files being *created*? That shouldn’t be stream file objects,
it’s either a filter above you bypassing you or broken logic in your driver.
Can you see these files being created with FileSpy?

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntfsd…
> I’m tracking filenames in a filter driver, hooking into IRP_MJ_CREATE.
> This is not always called and I’m missing the creation of a bunch of
> files. Looks to me like this may be caused by IoCreateStreamFileObject
> bypassing the IRP_MJ_CREATE. Anyone got any suggestions on how to get the
> missing filename data?
>
>

3

Thanks Scott.

In a previous thread, Rod W posted…

“One thing, which is slightly orthogonal. If you are on top of NTFS you will see paging IO for files on file objects for which you have never see the create. So if (as most people do) you are attaching the StreamHandle context during post create and then you look only at paging writes you will be missing a bunch,”

This seems to be the bahaviour I’m seeing.

4

> "One thing, which is slightly orthogonal. If you are on top of NTFS you will see paging IO for files on

file objects for which you have never see the create

The word “create” in Windows FSDs have 2 meaning - a) MJ_CREATE path, i.e. any file creation/open and b) actual on-disk file creation, which is a subkind of MJ_CREATE.

Stream file objects can bypass a) but not b).

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

5

> > "One thing, which is slightly orthogonal. If you are on top of NTFS

you will see paging IO for files on
>file objects for which you have never see the create

The word “create” in Windows FSDs have 2 meaning - a) MJ_CREATE path,
i.e. any file creation/open and b) actual on-disk file creation, which
is a subkind of MJ_CREATE.

Stream file objects can bypass a) but not b).

Maxim, how would a stream file object bypass a?

Thanks
Doug

6

The filesystem might need to open a FILE_OBJECT to a stream that is stored
on the volume (USN journal, MFT and so on). In theory it could simply go
ahead and read and write disk sectors because it always must know where they
are. However, it might need to access them a lot (like the MFT) so it makes
sense to cache them. Now, should the file system write its internal stream
caching mechanism or should it use the OS one (provided that the operating
system supports this) ? The decision for NTFS was to use the OS cache (the
Cache manager) for this. This means that NTFS must now conform to the rules
specified CC interfaces. Some CC interfaces require a FILE_OBJECT to be
passed in to identify the stream (look at CcCanIWrite for example). So now
NTFS must create FILE_OBJECTs for the internal streams it wants to use so
that it can call the CC interface and get the super smart caching it
provides. Again, it has the choice to go through the IO manager and have it
create a FILE_OBJECT for the stream it wants or to simply allocate a
FILE_OBJECT structure, initialize the fields it needs (it can do that
because it is on both ends of the request so it knows exactly what it needs)
and use that with CC. NTFS implemented the later approach in some cases.
These FILE_OBJECTs are stream file objects and they can be created by
calling IoCreateStreamFileObject. Since these objects are allocated by the
IO manager and then immediately returned to the file system, there is no
IRP_MJ_CREATE for them since the file system already knows which stream they
must be associated with. However, they will be seen in IO operations because
CC will treat them exactly as regular FILE_OBJECTs and issue IRPs and such
for them.

Now, one thing to note is that all the OS modules that will interact with
the FILE_OBJECT must be given a chance to initialize their private fields
associated with it. OB will be called to allocate the object so it will do
it there, NTFS will obviously do this immediately it gets the FILE_OBJECT,
CC will do it while setting up the cache but IO manager must be given a
chance to do it as well since it will see the FILE_OBJECT at times (as I
said before CC will issue IRPs which IO manager will be involved with…).
So it means that the fields in the FILE_OBJECT that are meaningful for the
IO manager must be initialized as well. So either NTFS will do it (which is
a bad idea since it means NTFS must be aware of IO manager’s internals) or
simply notify IO manager that it should initialize this FILE_OBJECT. And
since NTFS must call some IO manager API then, why now have IO manager
allocate the FILE_OBJECT as well, since it has a lookaside list already and
it can cache FILE_OBJECTs better… So this is why IoCreateStreamFileObject
is what is used to create stream file objects.

Does this answer your question ?

Thanks,
Alex.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doug
Sent: Friday, November 05, 2010 6:42 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing IRP_MJ_CREATE

Maxim, how would a stream file object bypass a?

Thanks
Doug

7

That answers my question.

Does that mean that only internal OS file access can occur without
seeing a IRP_MJ_CREATE go by (ie only journal, MFT, etc)?

(And apologies to the OP for the hijack if this wasn’t something
he was going to ask)

Doug

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-430740-
xxxxx@lists.osr.com] On Behalf Of Alex Carp
Sent: Friday, November 05, 2010 10:48 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing IRP_MJ_CREATE

The filesystem might need to open a FILE_OBJECT to a stream that is
stored
on the volume (USN journal, MFT and so on). In theory it could simply
go
ahead and read and write disk sectors because it always must know where
they
are. However, it might need to access them a lot (like the MFT) so it
makes
sense to cache them. Now, should the file system write its internal
stream
caching mechanism or should it use the OS one (provided that the
operating
system supports this) ? The decision for NTFS was to use the OS cache
(the
Cache manager) for this. This means that NTFS must now conform to the
rules
specified CC interfaces. Some CC interfaces require a FILE_OBJECT to be
passed in to identify the stream (look at CcCanIWrite for example). So
now
NTFS must create FILE_OBJECTs for the internal streams it wants to use
so
that it can call the CC interface and get the super smart caching it
provides. Again, it has the choice to go through the IO manager and
have it
create a FILE_OBJECT for the stream it wants or to simply allocate a
FILE_OBJECT structure, initialize the fields it needs (it can do that
because it is on both ends of the request so it knows exactly what it
needs)
and use that with CC. NTFS implemented the later approach in some
cases.
These FILE_OBJECTs are stream file objects and they can be created by
calling IoCreateStreamFileObject. Since these objects are allocated by
the
IO manager and then immediately returned to the file system, there is
no
IRP_MJ_CREATE for them since the file system already knows which stream
they
must be associated with. However, they will be seen in IO operations
because
CC will treat them exactly as regular FILE_OBJECTs and issue IRPs and
such
for them.

Now, one thing to note is that all the OS modules that will interact
with
the FILE_OBJECT must be given a chance to initialize their private
fields
associated with it. OB will be called to allocate the object so it will
do
it there, NTFS will obviously do this immediately it gets the
FILE_OBJECT,
CC will do it while setting up the cache but IO manager must be given a
chance to do it as well since it will see the FILE_OBJECT at times (as
I
said before CC will issue IRPs which IO manager will be involved
with…).
So it means that the fields in the FILE_OBJECT that are meaningful for
the
IO manager must be initialized as well. So either NTFS will do it
(which is
a bad idea since it means NTFS must be aware of IO manager’s internals)
or
simply notify IO manager that it should initialize this FILE_OBJECT.
And
since NTFS must call some IO manager API then, why now have IO manager
allocate the FILE_OBJECT as well, since it has a lookaside list already
and
it can cache FILE_OBJECTs better… So this is why
IoCreateStreamFileObject
is what is used to create stream file objects.

Does this answer your question ?

Thanks,
Alex.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doug
Sent: Friday, November 05, 2010 6:42 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Missing IRP_MJ_CREATE

Maxim, how would a stream file object bypass a?

Thanks
Doug

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

8

Actually, I don’t know. I’d be interested to find out as well. I’m sure some
of the other contributors to the list can answer so let’s see :).

Thanks,
Alex.

9

> Does this answer your question ?

Great text! maybe even better then Rajeev’s book.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

10

> Does this answer your question ?

Alex’s posts are always informative, but this one is probably one of the best I’ve seen on this list in a while. Great contribution!

Thanks!

11

> Does that mean that only internal OS file access can occur without

seeing a IRP_MJ_CREATE go by (ie only journal, MFT, etc)?

In my experience, no.

There are some filter drivers (including but not limited to antivirus/security product) who perform file access by sending an IRP_MJ_CREATE to the bottom of the filters stack (directly to the file system driver). You won’t get to easily see those.

Razvan

12

>>There are some filter drivers (including but not limited to antivirus/security product) who perform file access by sending an IRP_MJ_CREATE to the bottom of the filters stack (directly to the file system driver).

Same with some virtualization software but there is no need to worry about that as these softwares understand the cost of bypassing the stack. Generic apps don’t.

Gr8 post Alex.

13

Thanks for all the positive feedback everyone ;).

Regards,
Alex.