When is a IRP_MJ_SET_INFORMATION request issued?

NTFSD
1

When someone wishes to update a file, that would increase the size of the
file, when would IRP_MJ_SET_INFORMATION be called? would it be called
before data is physically written to the media or after to update the
filesize on the on disk directory entry?

the reason why i ask is because when looking through the fastfat example, it
appears that IRP_MJ_SET_INFORMATION allocates additional space on the disk,
which suggests that this request is sent before IRP_MJ_WRITE…

however if you monitor the filesystem activity when updating a file through
notepad, IRP_MJ_WRITE is actually sent before IRP_MJ_SET_INFORMATION… then
sortly after IRP_MJ_WRITE and IRP_MJ_SET_INFORMATION is called again… which
left me kinda confused…

I think whats happening here is on the first IRP_MJ_WRITE is a cache write
operation, IRP_MJ_SET_INFORMATION is called to update the file size and if
needed allocate additional disk space, the second IRP_MJ_WRITE is a non
cached io operation, which physically writes the data to the disk… I’ve no
idea why there is another IRP_MJ_SET_INFORMATION request tho!!

whats actually happening here? should IRP_MJ_SET_INFORMATION be called
before data is physically written to disk?

Regards
James Dunning

General Dynamics United Kingdom Limited
Registered in England and Wales No. 1911653
Registered Office: 100 New Bridge Street, London, EC4V 6JA

2

> When someone wishes to update a file, that would increase the size
of the

file, when would IRP_MJ_SET_INFORMATION be called? would it be
called
before data is physically written to the media or after to update
the
filesize on the on disk directory entry?

It will not be called at all. The size will be increased in the write
path.

the reason why i ask is because when looking through the fastfat
example, it
appears that IRP_MJ_SET_INFORMATION allocates additional space on
the disk,
which suggests that this request is sent before IRP_MJ_WRITE…

No, write also does this.

I think whats happening here is on the first IRP_MJ_WRITE is a cache
write
operation, IRP_MJ_SET_INFORMATION is called to update the file size
and if

IIRC this is the following:

  • IRP_MJ_WRITE to cached path
  • Cc function is called to update the file size
  • Cc sends IRP_MJ_SET_INFORMATION
  • IRP_MJ_SET_INFORMATION to the FSD

needed allocate additional disk space, the second IRP_MJ_WRITE is a
non
cached io operation, which physically writes the data to the disk…

Non-top-level (i.e. originated from Cc) IO operations must never
change ValidDataLength. Another rule is that paging IO (both
originated from Cc or from user mapping) must never change the file
size. So, writes sent by Cc must not change any file sizes.

So, at the moment of this non-top-level write, you must already have
allocated runs or clusters for the growing file.

This is extremely subtle thing. Read FASTFAT source. I would even
suggest copy-pasting from it. Also this thing is discussed in Rajeev
Nagar’s book.

Max

3

I have just double checked the IRP_MJ_WRITE dispatch handler, and you were
right, IRP_MJ_WRITE does allocate additional space on the disk if the file
is being
“extended”

but why does IRP_MJ_SET_INFORMATION also allocate additional space?

Is this so that the user can extend the physical size of a file, by
allocating additional
space, without writing any contents to the file? apart from of course the
filesize in
the directory entry?

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: 17 September 2002 22:43
To: File Systems Developers
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

When someone wishes to update a file, that would increase the size
of the
file, when would IRP_MJ_SET_INFORMATION be called? would it be
called
before data is physically written to the media or after to update
the
filesize on the on disk directory entry?

It will not be called at all. The size will be increased in the write
path.

the reason why i ask is because when looking through the fastfat
example, it
appears that IRP_MJ_SET_INFORMATION allocates additional space on
the disk,
which suggests that this request is sent before IRP_MJ_WRITE…

No, write also does this.

I think whats happening here is on the first IRP_MJ_WRITE is a cache
write
operation, IRP_MJ_SET_INFORMATION is called to update the file size
and if

IIRC this is the following:

  • IRP_MJ_WRITE to cached path
  • Cc function is called to update the file size
  • Cc sends IRP_MJ_SET_INFORMATION
  • IRP_MJ_SET_INFORMATION to the FSD

needed allocate additional disk space, the second IRP_MJ_WRITE is a
non
cached io operation, which physically writes the data to the disk…

Non-top-level (i.e. originated from Cc) IO operations must never
change ValidDataLength. Another rule is that paging IO (both
originated from Cc or from user mapping) must never change the file
size. So, writes sent by Cc must not change any file sizes.

So, at the moment of this non-top-level write, you must already have
allocated runs or clusters for the growing file.

This is extremely subtle thing. Read FASTFAT source. I would even
suggest copy-pasting from it. Also this thing is discussed in Rajeev
Nagar’s book.

Max

You are currently subscribed to ntfsd as:
xxxxx@generaldynamics.uk.com
To unsubscribe send a blank email to %%email.unsub%%

General Dynamics United Kingdom Limited
Registered in England and Wales No. 1911653
Registered Office: 100 New Bridge Street, London, EC4V 6JA

4

Yes, it is possible to extend the file size this way. The FS will however
treat any data after the last byte written as invalid, or zeroed, though,
since it is undefined.
To change the size via IRP_MJ_SET_INFORMATION the user is to call
SetFilePointer followed by SetEndOfFile. I think the Explorer copies files
this way.
The code in IRP_MJ_WRITE also performs extend checks, since the user might
just write the data without extending the file, first.

Regards, Dejan.

James Dunning wrote:

I have just double checked the IRP_MJ_WRITE dispatch handler, and you were
right, IRP_MJ_WRITE does allocate additional space on the disk if the file
is being “extended”

but why does IRP_MJ_SET_INFORMATION also allocate additional space?

Is this so that the user can extend the physical size of a file, by
allocating additional space, without writing any contents to the file? apart
from of course the filesize in the directory entry?

–
Kind regards, Dejan M. www.alfasp.com
E-mail: xxxxx@alfasp.com ICQ#: 56570367
Alfa File Monitor - File monitoring library for Win32 developers.
Alfa File Protector - File protection and hiding library for Win32 developers.

5

When someone wishes to update a file, that would increase the size of the
file, when would IRP_MJ_SET_INFORMATION be called? would it be called
before data is physically written to the media or after to update the
filesize on the on disk directory entry?

the reason why i ask is because when looking through the fastfat example, it
appears that IRP_MJ_SET_INFORMATION allocates additional space on the disk,
which suggests that this request is sent before IRP_MJ_WRITE…

however if you monitor the filesystem activity when updating a file through
notepad, IRP_MJ_WRITE is actually sent before IRP_MJ_SET_INFORMATION… then
sortly after IRP_MJ_WRITE and IRP_MJ_SET_INFORMATION is called again… which
left me kinda confused…

I think whats happening here is on the first IRP_MJ_WRITE is a cache write
operation, IRP_MJ_SET_INFORMATION is called to update the file size and if
needed allocate additional disk space, the second IRP_MJ_WRITE is a non
cached io operation, which physically writes the data to the disk… I’ve no
idea why there is another IRP_MJ_SET_INFORMATION request tho!!

whats actually happening here? should IRP_MJ_SET_INFORMATION be called
before data is physically written to disk?

Regards
James Dunning

General Dynamics United Kingdom Limited
Registered in England and Wales No. 1911653
Registered Office: 100 New Bridge Street, London, EC4V 6JA

6

This depends. If the user calls SetFilePointer followed by SetEndOfFile
BEFORE the write, the IRP_MJ_SET_INFORMATION / FileEndOfFileInformation will be
called before the write takes place.
However, the cache manager will call it after the non cached I/O, too.

Regards, Dejan.

James Dunning wrote:

When someone wishes to update a file, that would increase the size of the
file, when would IRP_MJ_SET_INFORMATION be called? would it be called
before data is physically written to the media or after to update the
filesize on the on disk directory entry?

the reason why i ask is because when looking through the fastfat example, it
appears that IRP_MJ_SET_INFORMATION allocates additional space on the disk,
which suggests that this request is sent before IRP_MJ_WRITE…

however if you monitor the filesystem activity when updating a file through
notepad, IRP_MJ_WRITE is actually sent before IRP_MJ_SET_INFORMATION… then
sortly after IRP_MJ_WRITE and IRP_MJ_SET_INFORMATION is called again… which
left me kinda confused…

I think whats happening here is on the first IRP_MJ_WRITE is a cache write
operation, IRP_MJ_SET_INFORMATION is called to update the file size and if
needed allocate additional disk space, the second IRP_MJ_WRITE is a non
cached io operation, which physically writes the data to the disk… I’ve no
idea why there is another IRP_MJ_SET_INFORMATION request tho!!

whats actually happening here? should IRP_MJ_SET_INFORMATION be called
before data is physically written to disk?

Regards
James Dunning

General Dynamics United Kingdom Limited
Registered in England and Wales No. 1911653
Registered Office: 100 New Bridge Street, London, EC4V 6JA

You are currently subscribed to ntfsd as: xxxxx@alfasp.com
To unsubscribe send a blank email to %%email.unsub%%

–
Kind regards, Dejan M. www.alfasp.com
E-mail: xxxxx@alfasp.com ICQ#: 56570367
Alfa File Monitor - File monitoring library for Win32 developers.
Alfa File Protector - File protection and hiding library for Win32 developers.

7

I’m a newbie, so dont believe me, but when a copy operation is happening,
I think if the OS “knows” how big the output file is, it will to set EOF
before any writes, to ensure available disk space. Also, it happens if
the user program explicitly sets EOF. CC Does it according to the
previous comments in this thread.

gap

8

Doubts. Setting the EOF will trigger CcZeroData, and the same disk
blocks will be written twice - first with zeroes, then with real data.
I don’t think the OS will do such a disk bandwidth waste.

Max

----- Original Message -----
From: “Greg Pearce”
To: “File Systems Developers”
Sent: Saturday, September 21, 2002 8:31 AM
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

> I’m a newbie, so dont believe me, but when a copy operation is
happening,
> I think if the OS “knows” how big the output file is, it will to set
EOF
> before any writes, to ensure available disk space. Also, it happens
if
> the user program explicitly sets EOF. CC Does it according to the
> previous comments in this thread.
>
> gap
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to %%email.unsub%%
>

9

No, it happens differently - the Cc ‘probably’ treats data after EOF
as sparse, so no memory is allocated - but none is written at the time the
EOF is set, by the user.
When the flush, or close occur the data that was actually written is
written - other data is written as zeroed data. That’s my observation of
this situation.

Regards, Dejan.

“Maxim S. Shatskih” wrote:

Doubts. Setting the EOF will trigger CcZeroData, and the same disk
blocks will be written twice - first with zeroes, then with real data. I
don’t think the OS will do such a disk bandwidth waste.

–
Kind regards, Dejan M. www.alfasp.com
E-mail: xxxxx@alfasp.com ICQ#: 56570367
Alfa File Monitor - File monitoring library for Win32 developers.
Alfa File Protector - File protection and hiding library for Win32
developers.

10

Certainly in the case of FASTFAT whose on-disk structure doesn’t support the notion of valid data length, I have observed this wasteful paging behavior in my filters. In spite of this, setting the end of file before writing the file’s contents has a clear advantage in that the file is less likely to be fragmented if its disk space is allocated all at once instead of piecemeal.

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Sunday, September 22, 2002 8:23 AM
To: File Systems Developers
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Doubts. Setting the EOF will trigger CcZeroData, and the same disk
blocks will be written twice - first with zeroes, then with real data.
I don’t think the OS will do such a disk bandwidth waste.

Max

----- Original Message -----
From: “Greg Pearce”
To: “File Systems Developers”
Sent: Saturday, September 21, 2002 8:31 AM
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

> I’m a newbie, so dont believe me, but when a copy operation is
happening,
> I think if the OS “knows” how big the output file is, it will to set
EOF
> before any writes, to ensure available disk space. Also, it happens
if
> the user program explicitly sets EOF. CC Does it according to the
> previous comments in this thread.
>
> gap
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to %%email.unsub%%
>

—
You are currently subscribed to ntfsd as: xxxxx@inin.com
To unsubscribe send a blank email to %%email.unsub%%

11

Apologies for a pure curiosity question, but I thought that I remembered
that the NT native interface allowed one to express the desired size of a
file during the NtCreateFile (or ZwCreateFile) operation (which, if so, I’d
expect to be the mechanism used by, e.g., CopyFile to pre-set the output
file size without zero-filling it). And it’s not clear to me (any more,
anyway - I may just have forgotten this detail) why the absence of valid
data length support (e.g., in FASTFAT) would inhibit the use of such a
preallocation mechanism, as long as EOF were kept from getting ahead of the
sequentially-copied data.

Thanks,

  • bill

----- Original Message -----
From: “Fuller, Rob”
To: “File Systems Developers”
Sent: Monday, September 23, 2002 11:40 AM
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Certainly in the case of FASTFAT whose on-disk structure doesn’t support the
notion of valid data length, I have observed this wasteful paging behavior
in my filters. In spite of this, setting the end of file before writing the
file’s contents has a clear advantage in that the file is less likely to be
fragmented if its disk space is allocated all at once instead of piecemeal.

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Sunday, September 22, 2002 8:23 AM
To: File Systems Developers
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Doubts. Setting the EOF will trigger CcZeroData, and the same disk
blocks will be written twice - first with zeroes, then with real data.
I don’t think the OS will do such a disk bandwidth waste.

Max

----- Original Message -----
From: “Greg Pearce”
To: “File Systems Developers”
Sent: Saturday, September 21, 2002 8:31 AM
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

> I’m a newbie, so dont believe me, but when a copy operation is
happening,
> I think if the OS “knows” how big the output file is, it will to set
EOF
> before any writes, to ensure available disk space. Also, it happens
if
> the user program explicitly sets EOF. CC Does it according to the
> previous comments in this thread.
>
> gap
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to %%email.unsub%%
>

—
You are currently subscribed to ntfsd as: xxxxx@inin.com
To unsubscribe send a blank email to %%email.unsub%%

—
You are currently subscribed to ntfsd as: xxxxx@metrocast.net
To unsubscribe send a blank email to %%email.unsub%%

12

You would think AllocationSize is a logical way to implement this.
As it turns out, the behavior of setting FileAllocationInformation or specifying
the AllocationSize parameter to XxCreateFile is inconsistent between FASTFAT
and NTFS, particularly in the case where it is used as a mechanism
to reduce the size of the file. Curiously though, setting the AllocationSize
to zero does work consistently on both file systems, and is used by some
programs to truncate files. However, outside of this special case, I’ve never
seen AllocationSize used by any component, Microsoft or otherwise, in any other
fashion. Perhaps this is the reason WIN32’s CopyFile API sets FileEndOfFileInformation
instead of relying on the AllocationSize mechanism of XxCreateFile.

-----Original Message-----
From: Bill Todd [mailto:xxxxx@metrocast.net]
Sent: Tuesday, September 24, 2002 2:54 AM
To: File Systems Developers
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Apologies for a pure curiosity question, but I thought that I remembered
that the NT native interface allowed one to express the desired size of a
file during the NtCreateFile (or ZwCreateFile) operation (which, if so, I’d
expect to be the mechanism used by, e.g., CopyFile to pre-set the output
file size without zero-filling it). And it’s not clear to me (any more,
anyway - I may just have forgotten this detail) why the absence of valid
data length support (e.g., in FASTFAT) would inhibit the use of such a
preallocation mechanism, as long as EOF were kept from getting ahead of the
sequentially-copied data.

Thanks,

  • bill

----- Original Message -----
From: “Fuller, Rob”
To: “File Systems Developers”
Sent: Monday, September 23, 2002 11:40 AM
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Certainly in the case of FASTFAT whose on-disk structure doesn’t support the
notion of valid data length, I have observed this wasteful paging behavior
in my filters. In spite of this, setting the end of file before writing the
file’s contents has a clear advantage in that the file is less likely to be
fragmented if its disk space is allocated all at once instead of piecemeal.

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Sunday, September 22, 2002 8:23 AM
To: File Systems Developers
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Doubts. Setting the EOF will trigger CcZeroData, and the same disk
blocks will be written twice - first with zeroes, then with real data.
I don’t think the OS will do such a disk bandwidth waste.

Max

----- Original Message -----
From: “Greg Pearce”
To: “File Systems Developers”
Sent: Saturday, September 21, 2002 8:31 AM
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

> I’m a newbie, so dont believe me, but when a copy operation is
happening,
> I think if the OS “knows” how big the output file is, it will to set
EOF
> before any writes, to ensure available disk space. Also, it happens
if
> the user program explicitly sets EOF. CC Does it according to the
> previous comments in this thread.
>
> gap
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to %%email.unsub%%
>

—
You are currently subscribed to ntfsd as: xxxxx@inin.com
To unsubscribe send a blank email to %%email.unsub%%

—
You are currently subscribed to ntfsd as: xxxxx@metrocast.net
To unsubscribe send a blank email to %%email.unsub%%

—
You are currently subscribed to ntfsd as: xxxxx@inin.com
To unsubscribe send a blank email to %%email.unsub%%

13

By the way, I believe it is the TRUNCATE_EXISTING parameter to WIN32’s
CreateFile API that results in the behavior of specifying an AllocationSize of zero
to NtCreateFile.

-----Original Message-----
From: Fuller, Rob
Sent: Friday, September 27, 2002 11:16 AM
To: ‘File Systems Developers’
Subject: RE: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

You would think AllocationSize is a logical way to implement this.
As it turns out, the behavior of setting FileAllocationInformation or specifying
the AllocationSize parameter to XxCreateFile is inconsistent between FASTFAT
and NTFS, particularly in the case where it is used as a mechanism
to reduce the size of the file. Curiously though, setting the AllocationSize
to zero does work consistently on both file systems, and is used by some
programs to truncate files. However, outside of this special case, I’ve never
seen AllocationSize used by any component, Microsoft or otherwise, in any other
fashion. Perhaps this is the reason WIN32’s CopyFile API sets FileEndOfFileInformation
instead of relying on the AllocationSize mechanism of XxCreateFile.

-----Original Message-----
From: Bill Todd [mailto:xxxxx@metrocast.net]
Sent: Tuesday, September 24, 2002 2:54 AM
To: File Systems Developers
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Apologies for a pure curiosity question, but I thought that I remembered
that the NT native interface allowed one to express the desired size of a
file during the NtCreateFile (or ZwCreateFile) operation (which, if so, I’d
expect to be the mechanism used by, e.g., CopyFile to pre-set the output
file size without zero-filling it). And it’s not clear to me (any more,
anyway - I may just have forgotten this detail) why the absence of valid
data length support (e.g., in FASTFAT) would inhibit the use of such a
preallocation mechanism, as long as EOF were kept from getting ahead of the
sequentially-copied data.

Thanks,

  • bill

----- Original Message -----
From: “Fuller, Rob”
To: “File Systems Developers”
Sent: Monday, September 23, 2002 11:40 AM
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Certainly in the case of FASTFAT whose on-disk structure doesn’t support the
notion of valid data length, I have observed this wasteful paging behavior
in my filters. In spite of this, setting the end of file before writing the
file’s contents has a clear advantage in that the file is less likely to be
fragmented if its disk space is allocated all at once instead of piecemeal.

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Sunday, September 22, 2002 8:23 AM
To: File Systems Developers
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

Doubts. Setting the EOF will trigger CcZeroData, and the same disk
blocks will be written twice - first with zeroes, then with real data.
I don’t think the OS will do such a disk bandwidth waste.

Max

----- Original Message -----
From: “Greg Pearce”
To: “File Systems Developers”
Sent: Saturday, September 21, 2002 8:31 AM
Subject: [ntfsd] Re: When is a IRP_MJ_SET_INFORMATION request issued?

> I’m a newbie, so dont believe me, but when a copy operation is
happening,
> I think if the OS “knows” how big the output file is, it will to set
EOF
> before any writes, to ensure available disk space. Also, it happens
if
> the user program explicitly sets EOF. CC Does it according to the
> previous comments in this thread.
>
> gap
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to %%email.unsub%%
>

—
You are currently subscribed to ntfsd as: xxxxx@inin.com
To unsubscribe send a blank email to %%email.unsub%%

—
You are currently subscribed to ntfsd as: xxxxx@metrocast.net
To unsubscribe send a blank email to %%email.unsub%%

—
You are currently subscribed to ntfsd as: xxxxx@inin.com
To unsubscribe send a blank email to %%email.unsub%%

14

>seen AllocationSize used by any component, Microsoft or otherwise, in
any other

fashion.

Maybe MSSQLServer?

Max