IRP_MJ_WRITE and paging I/O at EOF

NTFSD
1

I always assumed that paging i/o writes out an entire page of a file, no
matter the file length. However, on closer look it appears that paging I/O
only writes out a partial page as the last page of a file.

Let’s say that the final page of a file only contains 32 valid bytes. The
dispatch routine of the paging IRP_MJ_WRITE contains Parameters.Write.Length =
0x1000 as expected. However, the completion routine contains
IoStatus.Information = 32. Is paging i/o writing out 0x1000 bytes or just 32
bytes???

Thanks for any clarification.

2

The FSD chops off the paging I/O writes at EOF. Of course, EOF may have
increased by the time the filesystem returns from the write request, so
if you’re a filter don’t make any assumptions about current EOF based on
this.

  • Nick Ryan

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Neil Weicher
Sent: Saturday, May 31, 2003 8:57 PM
To: File Systems Developers
Subject: [ntfsd] IRP_MJ_WRITE and paging I/O at EOF

I always assumed that paging i/o writes out an entire page of
a file, no matter the file length. However, on closer look
it appears that paging I/O only writes out a partial page as
the last page of a file.

Let’s say that the final page of a file only contains 32
valid bytes. The dispatch routine of the paging IRP_MJ_WRITE
contains Parameters.Write.Length = 0x1000 as expected.
However, the completion routine contains IoStatus.Information
= 32. Is paging i/o writing out 0x1000 bytes or just 32 bytes???

Thanks for any clarification.

You are currently subscribed to ntfsd as: xxxxx@nryan.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

3

Yes, but how does the FSD know where the EOF is? The IRP_MJ_SET_INFORMATION
doesn’t come until *after* the IRP_MJ_WRITE.

Subject: RE: IRP_MJ_WRITE and paging I/O at EOF
From: “Nick Ryan”
> Date: Sat, 31 May 2003 21:14:44 -0700
> X-Message-Number: 1
>
> The FSD chops off the paging I/O writes at EOF. Of course, EOF may have
> increased by the time the filesystem returns from the write request, so
> if you’re a filter don’t make any assumptions about current EOF based on
> this.
>
> - Nick Ryan

4

The file system knows the size of the file at all times. When the file is
first opened, it comes from the attributes of the file. The only things
that then cause it to change are:

  • A truncation event (overwrite, supersede)
  • A set information (ah, but not the one from the cache manager)
  • An extending write (which originates from outside the VM system, since the
    VM system doesn’t DO extending writes)

Thus, there’s no time when the file system does not know the current size of
the file, although as Nick points out there is no guarantee it will not
change - indeed, this is one reason the Memory Manager locks the file (the
AcquireFile Fast I/O function’s job in life) while using that file size - it
doesn’t want it to change until the file object/section object relationship
has been well established.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Neil Weicher [mailto:xxxxx@netlib.com]
Sent: Monday, June 02, 2003 6:08 AM
To: File Systems Developers
Subject: [ntfsd] RE: IRP_MJ_WRITE and paging I/O at EOF

Yes, but how does the FSD know where the EOF is? The IRP_MJ_SET_INFORMATION
doesn’t come until *after* the IRP_MJ_WRITE.

Subject: RE: IRP_MJ_WRITE and paging I/O at EOF
From: “Nick Ryan”
> Date: Sat, 31 May 2003 21:14:44 -0700
> X-Message-Number: 1
>
> The FSD chops off the paging I/O writes at EOF. Of course, EOF may have
> increased by the time the filesystem returns from the write request, so
> if you’re a filter don’t make any assumptions about current EOF based on
> this.
>
> - Nick Ryan

—
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

5

Tony,

Thanks for the reply. I assume you mean it knows the size from FCB->FileSize?
In that case, what is the purpose of the Set Information? If the FSD already
knows the file size, who is Set Information for?

Thanks again.

Neil

Subject: RE: IRP_MJ_WRITE and paging I/O at EOF
From: Tony Mason

The file system knows the size of the file at all times. When the file is
first opened, it comes from the attributes of the file. The only things
that then cause it to change are:

  • A truncation event (overwrite, supersede)
  • A set information (ah, but not the one from the cache manager)
  • An extending write (which originates from outside the VM system, since the
    VM system doesn’t DO extending writes)

Thus, there’s no time when the file system does not know the current size of
the file, although as Nick points out there is no guarantee it will not
change - indeed, this is one reason the Memory Manager locks the file (the
AcquireFile Fast I/O function’s job in life) while using that file size - it
doesn’t want it to change until the file object/section object relationship
has been well established.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

6

Neil,

An IRP_MJ_SET_INFORMATION request to the file system is a request to CHANGE
the existing file size, not to tell the file system how big the file is.

There is this oddity where the cache manager tells the file system that the
file must be “at least this big” but the file systems (for all intents and
purposes) ignore this request because the file is ALREADY big enough (I
often point to this specific case in file systems class, because
implementing the “logical” thing here leads to erroneous results.) But the
original rationale there is to preserve the VM system guarantees - NO
EXTENDING WRITES.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Neil Weicher [mailto:xxxxx@netlib.com]
Sent: Tuesday, June 03, 2003 6:50 AM
To: File Systems Developers
Subject: [ntfsd] RE: IRP_MJ_WRITE and paging I/O at EOF

Tony,

Thanks for the reply. I assume you mean it knows the size from
FCB->FileSize?
In that case, what is the purpose of the Set Information? If the FSD
already
knows the file size, who is Set Information for?

Thanks again.

Neil

Subject: RE: IRP_MJ_WRITE and paging I/O at EOF
From: Tony Mason

The file system knows the size of the file at all times. When the file is
first opened, it comes from the attributes of the file. The only things
that then cause it to change are:

  • A truncation event (overwrite, supersede)
  • A set information (ah, but not the one from the cache manager)
  • An extending write (which originates from outside the VM system, since the
    VM system doesn’t DO extending writes)

Thus, there’s no time when the file system does not know the current size of
the file, although as Nick points out there is no guarantee it will not
change - indeed, this is one reason the Memory Manager locks the file (the
AcquireFile Fast I/O function’s job in life) while using that file size - it
doesn’t want it to change until the file object/section object relationship
has been well established.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

7

A file size may be extended by PAGING_IO under the following conditions:

1 - IRP_MJ_SET_INFORMATION(FileEndOfFileInformation) w/ AdvanceOnly := TRUE.
This is what Tony refers to as the “big enough” request. In other words -
don’t truncate the file!

2 - IRP_MJ_WRITE, IRP_MN_MDL request can extend the file.

/ted

-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Tuesday, June 03, 2003 7:16 AM
To: File Systems Developers
Subject: [ntfsd] RE: IRP_MJ_WRITE and paging I/O at EOF

Neil,

An IRP_MJ_SET_INFORMATION request to the file system is a request to CHANGE
the existing file size, not to tell the file system how big the file is.

There is this oddity where the cache manager tells the file system that the
file must be “at least this big” but the file systems (for all intents and
purposes) ignore this request because the file is ALREADY big enough (I
often point to this specific case in file systems class, because
implementing the “logical” thing here leads to erroneous results.) But the
original rationale there is to preserve the VM system guarantees - NO
EXTENDING WRITES.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Neil Weicher [mailto:xxxxx@netlib.com]
Sent: Tuesday, June 03, 2003 6:50 AM
To: File Systems Developers
Subject: [ntfsd] RE: IRP_MJ_WRITE and paging I/O at EOF

Tony,

Thanks for the reply. I assume you mean it knows the size from
FCB->FileSize?
In that case, what is the purpose of the Set Information? If the FSD
already knows the file size, who is Set Information for?

Thanks again.

Neil

Subject: RE: IRP_MJ_WRITE and paging I/O at EOF
From: Tony Mason

The file system knows the size of the file at all times. When the file is
first opened, it comes from the attributes of the file. The only things
that then cause it to change are:

  • A truncation event (overwrite, supersede)
  • A set information (ah, but not the one from the cache manager)
  • An extending write (which originates from outside the VM system, since the
    VM system doesn’t DO extending writes)

Thus, there’s no time when the file system does not know the current size of
the file, although as Nick points out there is no guarantee it will not
change - indeed, this is one reason the Memory Manager locks the file (the
AcquireFile Fast I/O function’s job in life) while using that file size - it
doesn’t want it to change until the file object/section object relationship
has been well established.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntfsd as: xxxxx@livevault.com To unsubscribe
send a blank email to xxxxx@lists.osr.com

8

Actually, paging I/O may never extend file size under any circumstances.
It may extend valid data, however, in the case of a user writing to a
memory-mapped file. In this case, the paging I/O will originate from the
mapped page-writer and the FSD will allow VDL to be extended because #1.
the request isn’t from the lazy writer, and #2. the request is
top-level.

  • Nick Ryan

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ted Hess
Sent: Tuesday, June 03, 2003 7:00 AM
To: File Systems Developers
Subject: [ntfsd] RE: IRP_MJ_WRITE and paging I/O at EOF

A file size may be extended by PAGING_IO under the following
conditions:

1 - IRP_MJ_SET_INFORMATION(FileEndOfFileInformation) w/
AdvanceOnly := TRUE. This is what Tony refers to as the “big
enough” request. In other words - don’t truncate the file!

2 - IRP_MJ_WRITE, IRP_MN_MDL request can extend the file.

/ted

-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Tuesday, June 03, 2003 7:16 AM
To: File Systems Developers
Subject: [ntfsd] RE: IRP_MJ_WRITE and paging I/O at EOF

Neil,

An IRP_MJ_SET_INFORMATION request to the file system is a
request to CHANGE the existing file size, not to tell the
file system how big the file is.

There is this oddity where the cache manager tells the file
system that the file must be “at least this big” but the file
systems (for all intents and
purposes) ignore this request because the file is ALREADY big
enough (I often point to this specific case in file systems
class, because implementing the “logical” thing here leads to
erroneous results.) But the original rationale there is to
preserve the VM system guarantees - NO EXTENDING WRITES.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Neil Weicher [mailto:xxxxx@netlib.com]
Sent: Tuesday, June 03, 2003 6:50 AM
To: File Systems Developers
Subject: [ntfsd] RE: IRP_MJ_WRITE and paging I/O at EOF

Tony,

Thanks for the reply. I assume you mean it knows the size from
FCB->FileSize?
In that case, what is the purpose of the Set Information? If
the FSD already knows the file size, who is Set Information for?

Thanks again.

Neil

Subject: RE: IRP_MJ_WRITE and paging I/O at EOF
From: Tony Mason

The file system knows the size of the file at all times.
When the file is first opened, it comes from the attributes
of the file. The only things that then cause it to change are:

  • A truncation event (overwrite, supersede)
  • A set information (ah, but not the one from the cache manager)
  • An extending write (which originates from outside the VM
    system, since the VM system doesn’t DO extending writes)

Thus, there’s no time when the file system does not know the
current size of the file, although as Nick points out there
is no guarantee it will not change - indeed, this is one
reason the Memory Manager locks the file (the AcquireFile
Fast I/O function’s job in life) while using that file size -
it doesn’t want it to change until the file object/section
object relationship has been well established.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntfsd as: xxxxx@livevault.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You are currently subscribed to ntfsd as: xxxxx@nryan.com
To unsubscribe send a blank email to xxxxx@lists.osr.com