IRP_MJ_WRITE

NTFSD
21

Good, looking forward to hearing how this turned out for you.

Regarding your doubts, I’m hoping someone with more experience could
clear up a few other things. I have a couple of questions myself
regarding this…

Writing to a file from user land should be more expensive than writing
to one from our own IRP. Is this true? I mean, if implemented
correctly, the kernel mode way to write logs to a file should be a
beter approach than sending the logs to user land, and then again
writing them to the disk.

I was thinking a kernel thread has a higher priority than a usermode
thread, therefor, if a lot of little writes are happening - this would
block usermode threads, giving the appearance that the system is hanging
or slowing(single processor). Whereas, if a usermode thread is doing a
lot of little writes, this might be slower, but it wouldn’t give the
same appearance because the ‘slow’ thread is more so isolated and
wouldn’t affect the rest of the win32 sub system or kernel code. Thus
creating an illusion.

I’m also wondering if you declared your log buffer as paged or non-paged
and if that could affect your driver’s performance. If the log was paged
mem and flushed to disk, then accessed again that would cause a fault,
then when saving the data to a log file, that would result in a third
disk access. LOL, I think I need to re-read those chapters on the CM…

M

22

Dear Matt, (and others),

Hello. Well I did try out the tweaks as adviced by you, 256K MDL size works
well.

I have the folowing doubts though…

In my code I had created the file in DriverEntry itself using
ZwCreateFile(…), this avoided the reentrancy problem well. But now I need
to make this driver a bootload driver and attach to NTFS when it is loaded(
may be after this driver loads). Thus ofcourse the call to ZwCreateFile
fails in DriverEntry(…) as no NTFS drives are mounted at that time. Great
said me, and was about to implement the Shadow device technique of creating
files (discussed in several OSR papers), when something else cam to my mind.

Why not port the creation code in SpyFsNotification(…), which will get
called after C: is mounted. Well, ZwCreateFile(…) works in this case, and
is called only once, but the system becomes awefully slow after that, I
don’t know why?

Can you (and ofcourse any one else) explain this strange behaviour.

23

> In my code I had created the file in DriverEntry

itself using
ZwCreateFile(…), this avoided the reentrancy
problem well. But now I need
to make this driver a bootload driver and attach to
NTFS when it is loaded(
may be after this driver loads). Thus ofcourse the
call to ZwCreateFile
fails in DriverEntry(…) as no NTFS drives are
mounted at that time.

Accessing files at boot time has been asked recently
on NTDEV. See this thread for a solution:

http://www.osronline.com/showThread.cfm?link=69255

Why not port the creation code in
SpyFsNotification(…), which will get
called after C: is mounted. Well, ZwCreateFile(…)
works in this case, and
is called only once, but the system becomes awefully
slow after that, I
don’t know why?

Can you (and ofcourse any one else) explain this
strange behaviour.

The SpyFsNotification routine is “invoked whenever a
file system has either registered or unregistered
itself as an active file system.” (quote taken from
filespy). Do not confuse this with a volume being
mounted, as they are not the same thing.

In other words, you are trying to create a file in the
notification routine for a file system being
registered. This routine gets called for each of the
file systems registering as active in the system, NOT
as a result of a volume mount.

Besides, how can you be sure SpyFsNotification will
get called after C: is mounted? It might get, but not
for the file system which mounted C: volume.

You might not fully understand how the mount process
works. If this is the case, try reading “Mounting a
Volume” article from the IFS help file.

Regards,
Razvan

Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

24

Razvan,

The thread you pointed out was very helpful, I will try and implement the
thread approach as suggested. But I have a few doubts.

  1. This worker thread is also in my FSFD, then if I call ZwCreateFile from
    inside it, why won’t it get into the reentrancy problem?
  2. I still cannot understand the possible link between the notification
    routine of Filespy and system getting slower. Yes, my fundamentals about
    mounting is hazy ( I have read the article in OSR, but I am still a beginner
    and grasping these things take time), but it still has no relation between
    systems stalling and file creation there! Maybe the routine is called
    multiple times, but they are a finite number of times(Windbg shows me so),
    after that I do normal read operations through explorer and notepad, so no
    question of it getting invoked again.

Please explain these two doubts, if you have time.

Regards,

amitr0

25

adding to my last two questions:

as proposed by Razvan and others, i implemented the threded approach, where
in driverentrry i create a thread that keeps trying a zwcreatefile op and
once it suceeds, get s the FO from it using Ob* functions and then exits the
while loop and does a PsTerminateSystemThread so that the thread is no more
(as it is not needed again).

Thoug the thread is killed, the system activity becomes dead slow after
that, I cannot find out why. can someone help. Taskmanager shows some a lot
of CPU kernel times, I suspect something is going wrong in my thread.

26

> 1. This worker thread is also in my FSFD, then if I

call ZwCreateFile from
inside it, why won’t it get into the reentrancy
problem?

Reentrancy can happen when you, for instance, try to
call ZwCreateFile inside your IRP_MJ_CREATE handler.
Due to your call to ZwCreateFile, an IRP_MJ_CREATE Irp
will be created and it might be sent to the filter
stack you are in so that your handler is very likely
to be called again in the context of the same thread,
which will trigger another ZwCreateFile, which will
result in your handler being called again… and so on
until you blow the current thread’s stack and you get
into problems.

  1. I still cannot understand the possible link
    between the notification
    routine of Filespy and system getting slower. Yes,
    my fundamentals about
    mounting is hazy ( I have read the article in OSR,
    but I am still a beginner
    and grasping these things take time), but it still
    has no relation between
    systems stalling and file creation there!

I did not say that there is a link between your code
in SpyFsNotification and the system getting slower.
However, I still think that you should take that code
out of there, because I see no point for it being
there.

I don’t know why your system gets slower, but I can’t
make any guess without further details about what your
filter is doing.

Maybe the
routine is called
multiple times, but they are a finite number of
times(Windbg shows me so),
after that I do normal read operations through
explorer and notepad, so no
question of it getting invoked again.

I’m sorry but I didn’t get your point here.

Regards,
Razvan

Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

27

Hello Razvan,

Well, the issue about the system getting slower is resolved. Though I cannot
understand why it happened. I was running the code on VmWare WksStn 5.5 and
the results I discussed were from the virtualizer. However, the code runs
just fine on a real machine. I really cannot tell you why this is
happening.]

Best Regards,

amitr0

28

When you defragment your VMWare disk does your performance improve
inside the VM? Or does this happen if you run your VM against a real
disk drive (yes, you can do this although it is a non-standard
configuration.)

Regards,

Tony

Tony Mason

Consulting Partner

OSR Open Systems Resources, Inc.

http://www.osr.com

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of amitr0
Sent: Monday, April 03, 2006 4:28 AM
To: ntfsd redirect
Subject: Re: [ntfsd] IRP_MJ_WRITE

Hello Razvan,

Well, the issue about the system getting slower is resolved. Though I
cannot understand why it happened. I was running the code on VmWare
WksStn 5.5 and the results I discussed were from the virtualizer.
However, the code runs just fine on a real machine. I really cannot tell
you why this is happening.]

Best Regards,

amitr0
— Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17 You are currently subscribed
to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a
blank email to xxxxx@lists.osr.com

29

Tony,

May be you are correct, I haven’y defragged the virtual disk in some time.
Also, though I know that a real disk can be used by the virtualizer, I dont
use one due to lack of resources.

30

AFAIK scheduling (or dispatching as someone purportedly wants to call
it) is independent of whether the thread is kernel mode or not.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of MM
Sent: Thursday, March 30, 2006 5:35 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] IRP_MJ_WRITE

I was thinking a kernel thread has a higher priority than a usermode
thread

M

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@appstream.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

31

Dispatching = “the mechanism necessary to switch from one thread to
another thread”
Scheduling = “the policy of choosing threads to execute”

From a kernel scheduling perspective, only the IDLE threads are special
(they are “chosen” not scheduled.) The Ke functions don’t distinguish
between user and kernel threads for the purposes of either scheduling
(policy) or dispatching. Other parts of the OS can (and do) vary from
this behavior.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Satya Das
Sent: Monday, April 03, 2006 5:37 PM
To: ntfsd redirect
Subject: RE: [ntfsd] IRP_MJ_WRITE

AFAIK scheduling (or dispatching as someone purportedly wants to call
it) is independent of whether the thread is kernel mode or not.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of MM
Sent: Thursday, March 30, 2006 5:35 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] IRP_MJ_WRITE

I was thinking a kernel thread has a higher priority than a usermode
thread

M

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@appstream.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

32

Hi,

can some one through some light on the following entities:

  1. FILE_COPY_OPERATION_NOT flag.
  2. file object has a flag named FILE_COPY_PASTE. what is it used for?
  3. The flag FO_SEQUENTIAL_ONLY. what is it’s use…does it haveany
    significance with copy operations?
33

I’m not even sure what you’re saying with (1) and (2). Are you
referring to the FILE_COPY_STRUCTURED_STORAGE (a vestigial definition
left over from a pre-W2K implementation)? There is no FILE_CUT_PASTE
flag that I can find in ntifs.h.

(3) is easier - it merely indicates the caller has indicated a
sequential access pattern; this allows scheduling read-ahead since it
will be useful (for random access, read-ahead doesn’t help.)

Regards,

Tony

Tony Mason

Consulting Partner

OSR Open Systems Resources, Inc.

http://www.osr.com

Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006.

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of amitr0
Sent: Tuesday, April 04, 2006 2:55 AM
To: ntfsd redirect
Subject: Re: [ntfsd] IRP_MJ_WRITE

Hi,

can some one through some light on the following entities:

  1. FILE_COPY_OPERATION_NOT flag.
  2. file object has a flag named FILE_COPY_PASTE. what is it used for?
  3. The flag FO_SEQUENTIAL_ONLY. what is it’s use…does it haveany
    significance with copy operations?
    — Questions? First check the IFS FAQ at
    https://www.osronline.com/article.cfm?id=17 You are currently subscribed
    to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a
    blank email to xxxxx@lists.osr.com
34

Tony (and others),

I’m not even sure what you’re saying with (1) and (2). Are you referring
to the FILE_COPY_STRUCTURED_STORAGE (a vestigial definition> left over from
a pre-W2K implementation)? There is no FILE_CUT_PASTE flag that I can find
in ntifs.h.

I have the same flummoxed expression on my face also. Why I asked this
question was because of this OSR thread…

http://www.osronline.com/showThread.cfm?link=70029

and this one…

http://www.osronline.com/showThread.cfm?link=34399

In the second thread mentioned above, the guy looks very confident. Either
he is, what is shows, and M$ is hiding some flags from us in NTIFS.H, or
else the whole thing is a hoax.

The last one (point 3), it is from this thread…

http://www.osronline.com/showThread.cfm?link=1244

I have read some papers published by Ho Mun Chuen, in the past, I cannot
rememebr which one, maybe something on cache replacements. it was good, so I
thought, what he states, might have some value.

35

I suspect (although Jamey can correct me if I’m wrong,) that at least
for the first two comments this was his way of saying “go away and do
your homework”. There is no flag and sending someone to find it would
be a way of making them “go away”.

The third discussion relates to heuristics of detecting file copy. I
don’t think that the specific comment in that case is serious either
(the emoticon at the end of his sentence is a suggestion that this might
be the case.)

Regards,

Tony

Tony Mason

Consulting Partner

OSR Open Systems Resources, Inc.

http://www.osr.com

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of amitr0
Sent: Tuesday, April 04, 2006 4:17 AM
To: ntfsd redirect
Subject: Re: [ntfsd] IRP_MJ_WRITE

Tony (and others),

I’m not even sure what you’re saying with (1) and (2). Are you
referring to the FILE_COPY_STRUCTURED_STORAGE (a vestigial definition>
left over from a pre-W2K implementation)? There is no FILE_CUT_PASTE
flag that I can find in ntifs.h.

I have the same flummoxed expression on my face also. Why I asked this
question was because of this OSR thread…

http://www.osronline.com/showThread.cfm?link=70029

and this one…

http://www.osronline.com/showThread.cfm?link=34399

In the second thread mentioned above, the guy looks very confident.
Either he is, what is shows, and M$ is hiding some flags from us in
NTIFS.H, or else the whole thing is a hoax.

The last one (point 3), it is from this thread…

http://www.osronline.com/showThread.cfm?link=1244

I have read some papers published by Ho Mun Chuen, in the past, I cannot
rememebr which one, maybe something on cache replacements. it was good,
so I thought, what he states, might have some value. — Questions?
First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’ To unsubscribe send a blank email to xxxxx@lists.osr.com

36

Well Tony :frowning:

maybe you are right in all cases, and I was too deeply involved in the
threads to detect the humor in it (duuh…)

But Inaki might have a point there when he says …

“However when using CopyFileEx, for example, the copy operation is made in a
particular way not used in every other copy method (although it might be
used by any application, of course). CopyFileEx maps the entire source file
and then writes to target file using a write with mapped memory as source
buffer. Also it creates the target file in exclusive mode (i.e.not shared
writes or reads are allowed until closed). This behaviour is a good clue to
guess this is a copy operation.”

Though, the better approah looks like hooking CopyFileEx instead.

37

>Writing to a file from user land should be more expensive than writing to

one from our own IRP. Is this true? I mean, if implemented correctly, the

Correct, but you must implement this “writing to a file” correctly in the
kernel mode, which seems to be not the case.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

38

>I suspect (although Jamey can correct me if I’m wrong,) that at least

for the first two comments this was his way of saying “go away and do
your homework”. There is no flag and sending someone to find it would
be a way of making them “go away”.

Surely there is no such flag, since there is no such facility in CreateFile.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com