Filter manager and cached operations

I’d like verification on something. Recently I had to change my filter so
some read operations are caught before going through the cache manager.
Trying a simple experiment from the command line of “type file” I do see a
byte offset of zero for all the read requests. The file was opened with
FILE_SEQUENTIAL_ONLY but the documentation on IRP_MJ_READ indicates that if
the current position was to be used, that either a null or a “special
value” should have been in the byte offset.

Is this a filter manager bug since it does not use the “special value” (it
obviously cannot be NULL) or a problem with documentation, or my brain
damage?


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

My typo, the file was opened with FILE_SYNCHRONOUS_IO_NONALERT. I think
the zero should be treated as use current file position, but then what
happens someone really wanted zero?


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

“Don Burn” wrote in message news:xxxxx@ntfsd…
> I’d like verification on something. Recently I had to change my filter
> so some read operations are caught before going through the cache
> manager. Trying a simple experiment from the command line of “type file”
> I do see a byte offset of zero for all the read requests. The file was
> opened with FILE_SEQUENTIAL_ONLY but the documentation on IRP_MJ_READ
> indicates that if the current position was to be used, that either a null
> or a “special value” should have been in the byte offset.
>
> Is this a filter manager bug since it does not use the “special value”
> (it obviously cannot be NULL) or a problem with documentation, or my
> brain damage?
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> http://www.windrvr.com
> Remove StopSpam from the email to reply
>
>
>
>

My recollection is that FILE_SEQUENTIAL_ONLY is simply a hint to to cache
manager. It has no effect on the interpretation of parameters like byte
offset, nor does it restrict the operations you can do. You’re free to
access the file non-sequentially, but will get less than optimal cache
behavior.

Offset 0 means offset 0. Full stop. A null offset is a non-sequitor,
because the offset, and not a pointer to it is stored in IrpSp->Parameters.
The “special” value FILE_USE_FILE_POINTER_POSITION (-2) indicates that the
saved position in file object should be used.

Are your type commands on files small enough that they may be reading the
entire file in a single I/O? In that case you would always see a zero
offset.

  • Dan.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Wednesday, January 24, 2007 3:35 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Filter manager and cached operations

My typo, the file was opened with FILE_SYNCHRONOUS_IO_NONALERT. I think
the zero should be treated as use current file position, but then what
happens someone really wanted zero?


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting http://www.windrvr.com
Remove StopSpam from the email to reply

“Don Burn” wrote in message news:xxxxx@ntfsd…
> I’d like verification on something. Recently I had to change my
> filter
> so some read operations are caught before going through the cache
> manager. Trying a simple experiment from the command line of “type file”
> I do see a byte offset of zero for all the read requests. The file was
> opened with FILE_SEQUENTIAL_ONLY but the documentation on IRP_MJ_READ
> indicates that if the current position was to be used, that either a null
> or a “special value” should have been in the byte offset.
>
> Is this a filter manager bug since it does not use the “special value”
> (it obviously cannot be NULL) or a problem with documentation, or my
> brain damage?
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> http://www.windrvr.com
> Remove StopSpam from the email to reply
>
>
>
>


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@privtek.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Isn’t this the same as for a legacy filter? (It’s 1 am so I might need coffee)

Don Burn wrote:

My typo, the file was opened with FILE_SYNCHRONOUS_IO_NONALERT. I think
the zero should be treated as use current file position, but then what
happens someone really wanted zero?


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

“Don Burn” wrote in message news:xxxxx@ntfsd…
> > I’d like verification on something. Recently I had to change my filter
> > so some read operations are caught before going through the cache
> > manager. Trying a simple experiment from the command line of “type file”
> > I do see a byte offset of zero for all the read requests. The file was
> > opened with FILE_SEQUENTIAL_ONLY but the documentation on IRP_MJ_READ
> > indicates that if the current position was to be used, that either a null
> > or a “special value” should have been in the byte offset.
> >
> > Is this a filter manager bug since it does not use the “special value”
> > (it obviously cannot be NULL) or a problem with documentation, or my
> > brain damage?
> >
> >
> > –
> > Don Burn (MVP, Windows DDK)
> > Windows 2k/XP/2k3 Filesystem and Driver Consulting
> > http://www.windrvr.com
> > Remove StopSpam from the email to reply
> >
> >
> >
> >
>
> —
> Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@alfasp.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com


King regards, Dejan
http://www.alfasp.com
File system audit, security and encryption kits.

From the WDK documentation:

IrpSp->Parameters.Read.ByteOffset
Pointer to a LARGE_INTEGER variable that specifies the starting byte offset
within the file of the data to be read.
Under certain circumstances, this parameter might validly contain a NULL or
negative value. For example, if the file object was opened for synchronous
I/O, and one of the following conditions is true, this indicates that the
current file position should be used instead of an explicit file offset
value:

IrpSp->Parameters.Read.ByteOffset == NULL

IrpSp->Parameters.Read.ByteOffset.LowPart == FILE_USE_FILE_POINTER_POSITION
and IrpSp->Parameters.Read.ByteOffset.HighPart == -1.

So is the above incorrect or does the Filter Manager mess uo on the first
case and report zero.

My minifilter acts as a redirector in some cases, so in this case sends the
call to another system where it always reads the same blocks, even though
the open had the same options.


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

“Dejan Maksimovic” wrote in message news:xxxxx@ntfsd…
>
> Isn’t this the same as for a legacy filter? (It’s 1 am so I might need
> coffee)
>
> Don Burn wrote:
>
>> My typo, the file was opened with FILE_SYNCHRONOUS_IO_NONALERT. I think
>> the zero should be treated as use current file position, but then what
>> happens someone really wanted zero?
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> http://www.windrvr.com
>> Remove StopSpam from the email to reply
>>
>> “Don Burn” wrote in message news:xxxxx@ntfsd…
>> > I’d like verification on something. Recently I had to change my
>> > filter
>> > so some read operations are caught before going through the cache
>> > manager. Trying a simple experiment from the command line of “type
>> > file”
>> > I do see a byte offset of zero for all the read requests. The file
>> > was
>> > opened with FILE_SEQUENTIAL_ONLY but the documentation on IRP_MJ_READ
>> > indicates that if the current position was to be used, that either a
>> > null
>> > or a “special value” should have been in the byte offset.
>> >
>> > Is this a filter manager bug since it does not use the “special value”
>> > (it obviously cannot be NULL) or a problem with documentation, or my
>> > brain damage?
>> >
>> >
>> > –
>> > Don Burn (MVP, Windows DDK)
>> > Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> > http://www.windrvr.com
>> > Remove StopSpam from the email to reply
>> >
>> >
>> >
>> >
>>
>> —
>> Questions? First check the IFS FAQ at
>> https://www.osronline.com/article.cfm?id=17
>>
>> You are currently subscribed to ntfsd as: xxxxx@alfasp.com
>> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
> –
> King regards, Dejan
> http://www.alfasp.com
> File system audit, security and encryption kits.
>
>
>

From wdm.h (definition of IO_STACK_LOCATION):

//
// System service parameters for: NtReadFile
//

struct {
ULONG Length;
ULONG POINTER_ALIGNMENT Key;
LARGE_INTEGER ByteOffset;
} Read;

That doc section is full of it.

Filter manager’s callback data parameters is a copy of the IO_STACK_LOCATION
parameters, in the same format. If you are seeing a zero ByteOffset, that’s
exactly what was requested.

In my experience, you will rarely see FILE_USE_FILE_POINTER_POSITION unless
you create your own tests for it. I believe Win32 ReadFile either maintains
its own pointer, or gets it from the file object. Either way you end up
seeing explicit offsets.

  • Dan.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Wednesday, January 24, 2007 5:53 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Filter manager and cached operations

From the WDK documentation:

IrpSp->Parameters.Read.ByteOffset
Pointer to a LARGE_INTEGER variable that specifies the starting byte offset
within the file of the data to be read.
Under certain circumstances, this parameter might validly contain a NULL or
negative value. For example, if the file object was opened for synchronous
I/O, and one of the following conditions is true, this indicates that the
current file position should be used instead of an explicit file offset
value:

IrpSp->Parameters.Read.ByteOffset == NULL

IrpSp->Parameters.Read.ByteOffset.LowPart ==
IrpSp->FILE_USE_FILE_POINTER_POSITION
and IrpSp->Parameters.Read.ByteOffset.HighPart == -1.

So is the above incorrect or does the Filter Manager mess uo on the first
case and report zero.

My minifilter acts as a redirector in some cases, so in this case sends the
call to another system where it always reads the same blocks, even though
the open had the same options.


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting http://www.windrvr.com
Remove StopSpam from the email to reply

“Dejan Maksimovic” wrote in message news:xxxxx@ntfsd…
>
> Isn’t this the same as for a legacy filter? (It’s 1 am so I might
> need
> coffee)
>
> Don Burn wrote:
>
>> My typo, the file was opened with FILE_SYNCHRONOUS_IO_NONALERT. I
>> think the zero should be treated as use current file position, but
>> then what happens someone really wanted zero?
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> http://www.windrvr.com Remove StopSpam from the email to reply
>>
>> “Don Burn” wrote in message news:xxxxx@ntfsd…
>> > I’d like verification on something. Recently I had to change my
>> > filter
>> > so some read operations are caught before going through the cache
>> > manager. Trying a simple experiment from the command line of “type
>> > file”
>> > I do see a byte offset of zero for all the read requests. The file
>> > was
>> > opened with FILE_SEQUENTIAL_ONLY but the documentation on IRP_MJ_READ
>> > indicates that if the current position was to be used, that either a
>> > null
>> > or a “special value” should have been in the byte offset.
>> >
>> > Is this a filter manager bug since it does not use the “special
>> > value” (it obviously cannot be NULL) or a problem with
>> > documentation, or my brain damage?
>> >
>> >
>> > –
>> > Don Burn (MVP, Windows DDK)
>> > Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> > http://www.windrvr.com Remove StopSpam from the email to reply
>> >
>> >
>> >
>> >
>>
>> —
>> Questions? First check the IFS FAQ at
>> https://www.osronline.com/article.cfm?id=17
>>
>> You are currently subscribed to ntfsd as: xxxxx@alfasp.com To
>> unsubscribe send a blank email to xxxxx@lists.osr.com
>
> –
> King regards, Dejan
> http://www.alfasp.com
> File system audit, security and encryption kits.
>
>
>


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@privtek.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

HI Don

This NULL for ByteOffset business is a long time and well known error in the
documentation; it is instructive to compare this page with the page for
ZwReadFile :wink: You should check the header files because the field isnt a
pointer; ah I can see that Dan has pasted the relevant extract. Further I
think you’ll find that none of the sample file systems reference/implement
FILE_USE_FILE_POINTER_POSITION.

Cheers
Lyndon

“Don Burn” wrote in message news:xxxxx@ntfsd…
> From the WDK documentation:
>
> IrpSp->Parameters.Read.ByteOffset
> Pointer to a LARGE_INTEGER variable that specifies the starting byte
> offset within the file of the data to be read.
> Under certain circumstances, this parameter might validly contain a NULL
> or negative value. For example, if the file object was opened for
> synchronous I/O, and one of the following conditions is true, this
> indicates that the current file position should be used instead of an
> explicit file offset value:
>
> IrpSp->Parameters.Read.ByteOffset == NULL
>
> IrpSp->Parameters.Read.ByteOffset.LowPart ==
> FILE_USE_FILE_POINTER_POSITION and
> IrpSp->Parameters.Read.ByteOffset.HighPart == -1.
>
>
>
> So is the above incorrect or does the Filter Manager mess uo on the first
> case and report zero.
>
> My minifilter acts as a redirector in some cases, so in this case sends
> the call to another system where it always reads the same blocks, even
> though the open had the same options.
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> http://www.windrvr.com
> Remove StopSpam from the email to reply
>
>
>
>
>
> “Dejan Maksimovic” wrote in message news:xxxxx@ntfsd…
>>
>> Isn’t this the same as for a legacy filter? (It’s 1 am so I might need
>> coffee)
>>
>> Don Burn wrote:
>>
>>> My typo, the file was opened with FILE_SYNCHRONOUS_IO_NONALERT. I think
>>> the zero should be treated as use current file position, but then what
>>> happens someone really wanted zero?
>>>
>>> –
>>> Don Burn (MVP, Windows DDK)
>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>> http://www.windrvr.com
>>> Remove StopSpam from the email to reply
>>>
>>> “Don Burn” wrote in message news:xxxxx@ntfsd…
>>> > I’d like verification on something. Recently I had to change my
>>> > filter
>>> > so some read operations are caught before going through the cache
>>> > manager. Trying a simple experiment from the command line of “type
>>> > file”
>>> > I do see a byte offset of zero for all the read requests. The file
>>> > was
>>> > opened with FILE_SEQUENTIAL_ONLY but the documentation on IRP_MJ_READ
>>> > indicates that if the current position was to be used, that either a
>>> > null
>>> > or a “special value” should have been in the byte offset.
>>> >
>>> > Is this a filter manager bug since it does not use the “special value”
>>> > (it obviously cannot be NULL) or a problem with documentation, or my
>>> > brain damage?
>>> >
>>> >
>>> > –
>>> > Don Burn (MVP, Windows DDK)
>>> > Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>> > http://www.windrvr.com
>>> > Remove StopSpam from the email to reply
>>> >
>>> >
>>> >
>>> >
>>>
>>> —
>>> Questions? First check the IFS FAQ at
>>> https://www.osronline.com/article.cfm?id=17
>>>
>>> You are currently subscribed to ntfsd as: xxxxx@alfasp.com
>>> To unsubscribe send a blank email to xxxxx@lists.osr.com
>>
>> –
>> King regards, Dejan
>> http://www.alfasp.com
>> File system audit, security and encryption kits.
>>
>>
>>
>
>
>

> Offset 0 means offset 0. Full stop. A null offset is a non-sequitor,

because the offset, and not a pointer to it is stored in IrpSp->Parameters.
The “special” value FILE_USE_FILE_POINTER_POSITION (-2) indicates that
the
saved position in file object should be used.

At ZwXxx syscall level:

  • for overlapped file, you MUST provide a non-NULL pointer to ByteOffset to
    ZwRead/WriteFile. Win32 takes this value from OVERLAPPED. The value can be
    zero, but this means &{0} and not NULL pointer.
  • for non-overlapped file (at this level FILE_SYNCHRONOUS_IO_xxx) - you CAN
    provide ByteOffset, but can omit it at all by providing NULL, which means -
    “use pointer position”.

At IRP level:

  • FILE_USE_POINTER_POSITION is used in IrpSp’s parameters if the ZwXxxFile call
    had NULL (omitted) ByteOffset.
  • otherwise, IrpSp has the same value as was provided to ZwXxxFile.
  • for non-overlapped file (FILE_SYNCHRONOUS_IO_xxx) - the lowest driver/FSD has
    the burden of adjusting FileObject->CurrentByteOffset as a result of the
    operation, regardless of how the byte offset was provided - explicitly or
    “pointer position”.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

> From the WDK documentation:

IrpSp->Parameters.Read.ByteOffset
Pointer to a LARGE_INTEGER variable that specifies the starting byte offset
within the file of the data to be read.

The word “pointer” is a doc err. It is a value, not a pointer.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Don,

I think we’ve pretty well hashed out the ByteOffset interpretation issue,
but probably not addressed your actual problem.

Is the issue that you expect to see cached reads for the entirety of the
file, and you are not? If so, you are forgetting about mapped sections.

I don’t know how the type command is implemented, but suppose it is a
CopyFileEx to the console. CopyFileEx maps the input file to a section, and
passes the address of the section to WriteFile. Because of this, you don’t
see cached reads. The cache manager’s section is mapped into the users
virtual address space, and reads are satisfied with paging I/O.

Hope this helps,

  • Dan.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Thursday, January 25, 2007 5:52 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Filter manager and cached operations

From the WDK documentation:

IrpSp->Parameters.Read.ByteOffset
Pointer to a LARGE_INTEGER variable that specifies the starting byte
offset
within the file of the data to be read.

The word “pointer” is a doc err. It is a value, not a pointer.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@privtek.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Dan,

Thanks for the help, after reading the various replies this morning, I
realized it was my braindamage. In the failing cases my filter is
basically becoming the file system since the call does not proceed down the
stack, and it is my responsibility to manage the FILE_OBJECT and the
CurrentByteOffset. At least I believe that is the case, and will be
working on that assumption today, unless somebody tells me I am really off.


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

“Dan Kyler” wrote in message news:xxxxx@ntfsd…
> Don,
>
> I think we’ve pretty well hashed out the ByteOffset interpretation issue,
> but probably not addressed your actual problem.
>
> Is the issue that you expect to see cached reads for the entirety of the
> file, and you are not? If so, you are forgetting about mapped sections.
>
> I don’t know how the type command is implemented, but suppose it is a
> CopyFileEx to the console. CopyFileEx maps the input file to a section,
> and
> passes the address of the section to WriteFile. Because of this, you
> don’t
> see cached reads. The cache manager’s section is mapped into the users
> virtual address space, and reads are satisfied with paging I/O.
>
> Hope this helps,
> - Dan.
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
> Sent: Thursday, January 25, 2007 5:52 AM
> To: Windows File Systems Devs Interest List
> Subject: Re:[ntfsd] Filter manager and cached operations
>
>
>> From the WDK documentation:
>>
>> IrpSp->Parameters.Read.ByteOffset
>> Pointer to a LARGE_INTEGER variable that specifies the starting byte
>> offset
>> within the file of the data to be read.
>
> The word “pointer” is a doc err. It is a value, not a pointer.
>
> –
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@privtek.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>

Don:

Yep, that appears to be correct. See read.c in FastFat example.

  • Dan.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Thursday, January 25, 2007 7:03 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Filter manager and cached operations

Dan,

Thanks for the help, after reading the various replies this morning, I
realized it was my braindamage. In the failing cases my filter is
basically becoming the file system since the call does not proceed down the
stack, and it is my responsibility to manage the FILE_OBJECT and the
CurrentByteOffset. At least I believe that is the case, and will be
working on that assumption today, unless somebody tells me I am really off.


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting http://www.windrvr.com
Remove StopSpam from the email to reply

“Dan Kyler” wrote in message news:xxxxx@ntfsd…
> Don,
>
> I think we’ve pretty well hashed out the ByteOffset interpretation
> issue, but probably not addressed your actual problem.
>
> Is the issue that you expect to see cached reads for the entirety of
> the file, and you are not? If so, you are forgetting about mapped
> sections.
>
> I don’t know how the type command is implemented, but suppose it is a
> CopyFileEx to the console. CopyFileEx maps the input file to a
> section, and passes the address of the section to WriteFile. Because
> of this, you don’t
> see cached reads. The cache manager’s section is mapped into the users
> virtual address space, and reads are satisfied with paging I/O.
>
> Hope this helps,
> - Dan.
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
> Shatskih
> Sent: Thursday, January 25, 2007 5:52 AM
> To: Windows File Systems Devs Interest List
> Subject: Re:[ntfsd] Filter manager and cached operations
>
>
>> From the WDK documentation:
>>
>> IrpSp->Parameters.Read.ByteOffset
>> Pointer to a LARGE_INTEGER variable that specifies the starting byte
>> offset within the file of the data to be read.
>
> The word “pointer” is a doc err. It is a value, not a pointer.
>
> –
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@privtek.com To
> unsubscribe send a blank email to xxxxx@lists.osr.com
>
>


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@privtek.com
To unsubscribe send a blank email to xxxxx@lists.osr.com