How to determine valid data length in paging IO

NTFSD
1

Hi,

My file system filter driver intercepts IRP_MJ_WRITE, then encrypt
user data. My problem is, if the file length is less than one page,
how can I know the valid length of the data to be paged to disk.
Irp->Parameters->Write.Length of course is not want I want.

And, I want to get this info before calling IoCallDriver in my IRP_MJ_WRITE
dispatch routine.

Thanks!

Chen Zemao

2

If you are making the file appear smaller than it actually is with your
filter, you should already know what the ‘false’ file length should be.
Else, how do you know what to return when the user queries for the file
length in the file query APIs?

  • Nicholas Ryan

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Zm Chen
Sent: Monday, March 24, 2003 8:16 AM
To: File Systems Developers
Subject: [ntfsd] How to determine valid data length in paging IO

Hi,

My file system filter driver intercepts IRP_MJ_WRITE, then
encrypt user data. My problem is, if the file length is less
than one page, how can I know the valid length of the data to
be paged to disk.
Irp->Parameters->Write.Length of course is not want I want.

And, I want to get this info before calling IoCallDriver in
my IRP_MJ_WRITE dispatch routine.

Thanks!

Chen Zemao

You are currently subscribed to ntfsd as: xxxxx@nryan.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

3

>My file system filter driver intercepts IRP_MJ_WRITE, then encrypt

user data. My problem is, if the file length is less than one page,
how can I know the valid length of the data to be paged to disk.
Irp->Parameters->Write.Length of course is not want I want.

And, I want to get this info before calling IoCallDriver in my
IRP_MJ_WRITE
dispatch routine.

FsControl field in FileObject points to FSRTL_COMMON_FCB_HEADER structure
that contains ValidDataLength. The only problem is without proper
syncronization it can change after you use the value and before it is used
by file system.

Alexei.

4

I thought that wasn’t possible during paging I/O? (The value change)

FsControl field in FileObject points to FSRTL_COMMON_FCB_HEADER structure
that contains ValidDataLength. The only problem is without proper
syncronization it can change after you use the value and before it is used
by file system.

–
Kind regards, Dejan M. MVP for DDK
http://www.alfasp.com E-mail: xxxxx@alfasp.com
Alfa Transparent File Encryptor - Transparent file encryption services.
Alfa File Protector - File protection and hiding library for Win32 developers.

Alfa File Monitor - File monitoring library for Win32 developers.

5

>I thought that wasn’t possible during paging I/O? (The value change)

PagingIOResource has to be acquired prior to sending IRP to prevent
changing due to concurrent activity while IRP didn’t reach file system.
In case of lazy writer it actually happens via cache manager callbacks. In
case of flushing cache it is done via FastIo AcquireForCcFlush. In other
cases it is probably not always true, at least FAT explicitly acquires
PagingIOResource during processing of paging writes apparently expecting
that it may not be acquired yet.

Alexei.

6

Exactly, you cannot acquire the PagingIoResource or a deadlock would
occur.
But, a file size change cannot occur during the paging write, only normal
write (append) (and of course SetInfo call).
The cache manager will surely prevent a SetInfo call from proceeding
before the write finishes.
MS?

>I thought that wasn’t possible during paging I/O? (The value change)

PagingIOResource has to be acquired prior to sending IRP to prevent
changing due to concurrent activity while IRP didn’t reach file system.
In case of lazy writer it actually happens via cache manager callbacks. In
case of flushing cache it is done via FastIo AcquireForCcFlush. In other
cases it is probably not always true, at least FAT explicitly acquires
PagingIOResource during processing of paging writes apparently expecting
that it may not be acquired yet.

–
Kind regards, Dejan M. MVP for DDK
http://www.alfasp.com E-mail: xxxxx@alfasp.com
Alfa Transparent File Encryptor - Transparent file encryption services.
Alfa File Protector - File protection and hiding library for Win32 developers.

Alfa File Monitor - File monitoring library for Win32 developers.

7

>But, a file size change cannot occur during the paging write, only normal

write (append) (and of course SetInfo call).
File size change cannot occur as the result of paging write, but it may
occur during paging write because of concurrent activity.
By the way, the original question was about ValidDataLength, it may change
as the result of paging write in case of memory mapped file.

The cache manager will surely prevent a SetInfo call from proceeding
before the write finishes.
MS?

Actually serialization of IO achived by acquiring resources. If
appropriate resource is already acquired when you are in filter you are
OK. Otherwise file system will provide additional syncronization
internally but it is too late for the filter.

Alexei.

8

I’m not sure whether the caller actually meant ‘ValidDataLength’ when he
said ‘valid length of the data’ - I believe he’s asking how, for a
paging I/O write that extends beyond EOF, to know where EOF is inside
that buffer at the time of the call. If he is ‘spoofing’ the size of the
file to userland, he’ll have to rely on his own concept of EOF (and
serialize changes to it using his own locks - technically a filesytem
filter no-no, but sometimes there is no other way).

  • Nicholas Ryan

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Alexei Jelvis
Sent: Monday, March 24, 2003 1:38 PM
To: File Systems Developers
Subject: [ntfsd] Re: How to determine valid data length in paging IO

>But, a file size change cannot occur during the paging write, only
>normal write (append) (and of course SetInfo call).
File size change cannot occur as the result of paging write,
but it may occur during paging write because of concurrent
activity. By the way, the original question was about
ValidDataLength, it may change as the result of paging write
in case of memory mapped file.

>The cache manager will surely prevent a SetInfo call from proceeding
>before the write finishes. MS?

Actually serialization of IO achived by acquiring resources.
If appropriate resource is already acquired when you are in
filter you are OK. Otherwise file system will provide
additional syncronization internally but it is too late for
the filter.

Alexei.

You are currently subscribed to ntfsd as: xxxxx@nryan.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

9

Hmm, I’m getting confused - how does a paging I/O extend the file size?
Or just PAGING_IO without NO_CACHE?

Nicholas Ryan wrote:

I’m not sure whether the caller actually meant ‘ValidDataLength’ when he
said ‘valid length of the data’ - I believe he’s asking how, for a
paging I/O write that extends beyond EOF, to know where EOF is inside
that buffer at the time of the call. If he is ‘spoofing’ the size of the
file to userland, he’ll have to rely on his own concept of EOF (and
serialize changes to it using his own locks - technically a filesytem
filter no-no, but sometimes there is no other way).

–
Kind regards, Dejan M. MVP for DDK
http://www.alfasp.com E-mail: xxxxx@alfasp.com
Alfa Transparent File Encryptor - Transparent file encryption services.
Alfa File Protector - File protection and hiding library for Win32
developers.
Alfa File Monitor - File monitoring library for Win32 developers.

10

Paging I/O doesn’t extend file size - I said ‘extends beyond EOF’, not
‘extends EOF’. :slight_smile:

  • Nicholas Ryan

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Dejan Maksimovic
Sent: Monday, March 24, 2003 4:22 PM
To: File Systems Developers
Subject: [ntfsd] Re: How to determine valid data length in paging IO

Hmm, I’m getting confused - how does a paging I/O extend
the file size? Or just PAGING_IO without NO_CACHE?

Nicholas Ryan wrote:

> I’m not sure whether the caller actually meant
‘ValidDataLength’ when
> he said ‘valid length of the data’ - I believe he’s asking
how, for a
> paging I/O write that extends beyond EOF, to know where EOF
is inside
> that buffer at the time of the call. If he is ‘spoofing’
the size of
> the file to userland, he’ll have to rely on his own concept of EOF
> (and serialize changes to it using his own locks - technically a
> filesytem filter no-no, but sometimes there is no other way).

–
Kind regards, Dejan M. MVP for DDK
http://www.alfasp.com E-mail: xxxxx@alfasp.com
Alfa Transparent File Encryptor - Transparent file encryption
services. Alfa File Protector - File protection and hiding
library for Win32 developers. Alfa File Monitor - File
monitoring library for Win32 developers.

You are currently subscribed to ntfsd as: xxxxx@nryan.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

11

Oh - my English is getting worse - or maybe it’s just my reading:-)

Nicholas Ryan wrote:

Paging I/O doesn’t extend file size - I said ‘extends beyond EOF’, not
‘extends EOF’. :slight_smile:

–
Kind regards, Dejan M. MVP for DDK
http://www.alfasp.com E-mail: xxxxx@alfasp.com
Alfa Transparent File Encryptor - Transparent file encryption services.
Alfa File Protector - File protection and hiding library for Win32
developers.
Alfa File Monitor - File monitoring library for Win32 developers.

12

> File size change cannot occur as the result of paging write, but it
may

occur during paging write because of concurrent activity.

Truncation cannot. PagingIoResource is for this.

Max

13

Sorry, I have not describe my question clearly.

My problem is about the valid data length in the last page to be paged.

So, ValidDataLength is not what I want.

Another question, the system hangs if I try to acquire PagingIOResource.
But, if I access the FsContext without acquiring PagingIOResource, it
hangs soon also.

Thanks!

Chen Zemao

14

Please tell us more about what you’re trying to do. I’m going to infer
that what you want to do is something like this:

FilterPagingIoWrite()
{
Perform some preprocessing on the request (maybe)
Send the write down to the filesystem, maybe in pieces
Perform some postprocessing on the request (maybe)
}

And you want to somehow guarantee that the file size (that you take out
of FsContext->FileSize - just accessing this field should not cause you
to deadlock) won’t change around this function.

Well, this isn’t really possible on a generic basis. If you KNOW you’re
working with a FastFat volume, you know that the file size will not
shrink during this call because the appropriate locks are already held,
but it may very well increase. Other filesystems may not even give you
this limited guarantee.

If you want to enforce synchronization of this code with file size
changes, you need to use your own locks to protect the file size on
files that your filter is interested in. Note that this is something
very dangerous for a filter to do, because you’re layering on top of an
existing locking mechanism that is already very complex. I suggest you
design your filter in such a way that you don’t have to take any locks
of your own during paging read/write operations - otherwise, deadlocks
are just too difficult to avoid no matter what filesystem you are
filtering.

  • Nicholas Ryan

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Zm Chen
Sent: Tuesday, March 25, 2003 6:41 PM
To: File Systems Developers
Subject: [ntfsd] RE: How to determine valid data length in paging IO

Sorry, I have not describe my question clearly.

My problem is about the valid data length in the last page to
be paged.

So, ValidDataLength is not what I want.

Another question, the system hangs if I try to acquire
PagingIOResource. But, if I access the FsContext without
acquiring PagingIOResource, it hangs soon also.

Thanks!

Chen Zemao

You are currently subscribed to ntfsd as: xxxxx@nryan.com
To unsubscribe send a blank email to xxxxx@lists.osr.com