Problems handling IRP_MJ_NETWORK_QUERY_OPEN

My HSM/reparse point minifilter needs to fake the size of a file when it
sees an IRP_MJ_NETWORK_QUERY_OPEN. I was working on this last week
without much luck. Details here:

http://www.osronline.com/showthread.cfm?link=131054

So I decided to see if I could circumvent the problem by disallowing
fast io for that operation. It wouldn’t be a long-term solution, but it
would let me verify that the incorrect file sizes coming back up are the
cause of the problems I’m seeing. When I disallow fastio, however, the
create that I see comes down with FILE_OPEN_REPARSE_POINT set. This
means that I don’t process the IRP in my IRP_MJ_QUERY_INFORMATION
postop. When I watch with procmon, the I see that the IRP gets
completed with “n/a” in the EndOfFile and AllocationSize fields.

Moral of the story, I can’t just disallow fastio on the file for this
operation. Would somebody like to hazard a guess as to what I might be
doing wrong for the NQO callbacks please?

What a giant pain in the ass for a simple problem.

~Eric

Most, if not all, of the existing HSM products mark the file sparse and zero out the data. This leaves the logical file size the same and eliminates the need to capture every Irp that can return size information. Maybe you have a good reason for not going this route but you should weigh it carefully against the extra work and potential performance impact of intercepting things like directory queries, etc. that return size info.

Just a suggestion.

Hmmm, thanks for suggesting that. It has been some extra work, and I’m
sure it’s a performance hit too (that I haven’t run up against in
testing - yet).

The good reason was “It seemed easier at the time”. Sigh.

I haven’t actually worked with sparse files before, so these might be
pretty basic questions.

Getting rid of the data looks pretty straightforward using
FSCTL_SET_SPARSE and FSCTL_SET_ZERO_DATA.

Getting the data back on restore, the file gets opened in kernel mode
through an interface my driver provides, and then data gets written in.
Once the data is all back, the RP gets deleted. Can I continue doing
this, except also unsetting the sparse attribute once it’s all back (if
it were unset to start with, that is. I probably shouldn’t unsparse a
file that was sparse, I can track that in my RP when I zero the file)?
The sparse file documentation says that it’s basically transparent to
userland processes, does that apply to kernel stuff too?

I guess the metaquestion is, how much work is it to change from what I’m
doing now?

~Eric

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@yahoo.com
Sent: Wednesday, May 14, 2008 6:44 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Problems handling IRP_MJ_NETWORK_QUERY_OPEN

Most, if not all, of the existing HSM products mark the file sparse and
zero out the data. This leaves the logical file size the same and
eliminates the need to capture every Irp that can return size
information. Maybe you have a good reason for not going this route but
you should weigh it carefully against the extra work and potential
performance impact of intercepting things like directory queries, etc.
that return size info.

Just a suggestion.


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@edsiohio.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

FSCTL_SET_SPARSE and FSCTL_SET_ZERO_DATA. >

Yes it is fairly easy. There are some gotcha’s - like the fact that marking a file sparse can actually make it consume more disk space. I have not seen any docs on this but I have seen that the physical size is rounded to a 16 cluster size. For instance if you take a 10 byte file (consuming 4K on disk if the cluster size if 4k) and make it sparse the physical size goes to 64K. If you are very near a disk full condition you may get an “out of disk space” error when trying to free up disk space. For very small files there is limited or no benefit.

through an interface my driver provides, and then data gets written in.
Once the data is all back, the RP gets deleted. Can I continue doing
this, except also unsetting the sparse attribute once it’s all back (if
it were unset to start with, that is. I probably shouldn’t unsparse a
file that was sparse, I can track that in my RP when I zero the file)?
The sparse file documentation says that it’s basically transparent to
userland processes, does that apply to kernel stuff too?>

NTFS does not allow you to remove the sparse attribute so once you mark it sparse it stays that way, although it does not necessarily mean that there are sparse regions in the file (see FSCTL_QUERY_ALLOCATED_REGIONS). It is transparent in userland. A user app can see the attribute and even determine what parts of the file have data but nothing special is needed to access and read the file.

doing now?>

I think in the long run you will find it much easier.