Problem recalling file on READ or AFSS

NTFSD
1

I currently have a successful HSM filter which recalls on Create. By
carefully inspecting flags etc, I have eliminated a great deal of spurious
recalls. In addition the user-mode agent keeps an eye on the file server so
ensuring that long recalls (stalling the IRP_MJ_CREATE) do not cause said
file server any major problems. My ‘stub’ file is a reparse point, with the
OFFLINE file attribute set and the primary data stream removed by making it
a sparse file with no actual data.

However I am now experimenting with recall on I/O (Read, Write,
AcquireForSectionSynchronization) and have hit a problem. I suspect I am
missing something fundamental, but doc reading, Googling, searching NTFSD
etc have all failed to illuminate it!

In (post) processing the Create when I see the reparse point for a file I
want to recall on I/O, I set some flags then reissue the IRP…

SetFlag (pData->Iopb->Parameters.Create.Options,
FILE_OPEN_REPARSE_POINT);
SetFlag (pData->Iopb->Parameters.Create.SecurityContext->DesiredAccess,
FILE_WRITE_ATTRIBUTES);
FltSetCallbackDataDirty (pData);
FltReissueSynchronousIo (pFltObjects->Instance, pData);

This gets the ‘stub’ file open so the caller gets his handle. I also
associate a StreamHandle context so I can detect this state in
pre-processing Read, AcquireForSectionSynchronization etc.

When I see the I/O for the file (detected by the StreamHandle context), I
use a worker thread to proces the recall, via FltAllocateGenericWorkItem. An
IRP can be pended, but FASTIO and FS_FILTER (as found in AFSS) is stalled
using
KeWaitForSingleObject (&pSHContext->sEvent, Executive, KernelMode,
FALSE, NULL);

In either case, when the worker thread calls FltWriteFile to repopulate the
file data, it blocks. Is this some kind of serialization? I was hoping that
this call could ‘overtake’ the Read or AFSS which was about to start. If it
cannot, how do you get around this? Do I have to use a different FileObject?
In which case do I then hit sharing problems with the user open?

Any light shed on this issue would be much appreciated.

By the way, I shall be at Plugfest#20, so if I need some significant
re-education beyond email and NTFSD, perhaps we can arrange that!

Thanks in anticipation,
John

2

Hey John,
A couple of points:

First of all, you can get the io manager to use the IRP path instead of
the FastIo path by returning FLT_PREOP_DISALLOW_FAST_IO. This will take
care of the fastio read/write. You only ever have to do this when the
data isn’t there, so if you’re recalling all the data at once, you’re
only doing this once if you’re concerned about disallowing fastio. It
seems that the consensus among people who know all of this better than I
do is that fastio isn’t the performance gain it once was anyhow.

For AFSS, I’ve written a second path in my code where I start the
recall, and then wait for an event. When the recall is done, I signal
the event, and the AFSS proceeds merrily along its way.

As far as FltWriteFile being blocked goes, I’m not sure why that’s
actually happening. Are you sending the write only to drivers below you
by setting the PFLT_INSTANCE correctly? I’ve never actually tried
writing the data using the same file object as the IO that initiated the
recall. Among other things, what happens if the handle wasn’t opened
with write access?

To get around this I would say use your own handle which you can open
with IO_IGNORE_SHARE_ACCESS_CHECK to prevent having share access issues.

Hope this gets you started.

I’ll be at the plugfest as well, by the way.

~Eric

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of John Leonard
Sent: Tuesday, January 06, 2009 5:29 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Problem recalling file on READ or AFSS

I currently have a successful HSM filter which recalls on
Create. By carefully inspecting flags etc, I have eliminated
a great deal of spurious recalls. In addition the user-mode
agent keeps an eye on the file server so ensuring that long
recalls (stalling the IRP_MJ_CREATE) do not cause said file
server any major problems. My ‘stub’ file is a reparse point,
with the OFFLINE file attribute set and the primary data
stream removed by making it a sparse file with no actual data.

However I am now experimenting with recall on I/O (Read, Write,
AcquireForSectionSynchronization) and have hit a problem. I
suspect I am missing something fundamental, but doc reading,
Googling, searching NTFSD etc have all failed to illuminate it!

In (post) processing the Create when I see the reparse point
for a file I want to recall on I/O, I set some flags then
reissue the IRP…

SetFlag (pData->Iopb->Parameters.Create.Options,
FILE_OPEN_REPARSE_POINT);
SetFlag
(pData->Iopb->Parameters.Create.SecurityContext->DesiredAccess,
FILE_WRITE_ATTRIBUTES);
FltSetCallbackDataDirty (pData);
FltReissueSynchronousIo (pFltObjects->Instance, pData);

This gets the ‘stub’ file open so the caller gets his handle.
I also associate a StreamHandle context so I can detect this
state in pre-processing Read, AcquireForSectionSynchronization etc.

When I see the I/O for the file (detected by the StreamHandle
context), I use a worker thread to proces the recall, via
FltAllocateGenericWorkItem. An IRP can be pended, but FASTIO
and FS_FILTER (as found in AFSS) is stalled using
KeWaitForSingleObject (&pSHContext->sEvent, Executive,
KernelMode, FALSE, NULL);

In either case, when the worker thread calls FltWriteFile to
repopulate the file data, it blocks. Is this some kind of
serialization? I was hoping that this call could ‘overtake’
the Read or AFSS which was about to start. If it cannot, how
do you get around this? Do I have to use a different FileObject?
In which case do I then hit sharing problems with the user open?

Any light shed on this issue would be much appreciated.

By the way, I shall be at Plugfest#20, so if I need some
significant re-education beyond email and NTFSD, perhaps we
can arrange that!

Thanks in anticipation,
John

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as:
xxxxx@edsiohio.com To unsubscribe send a blank email to
xxxxx@lists.osr.com

3

Are you seeing WriteFile stuck every time, or occasionally? Where exactly does it stuck (what thread stack looks like)?
In general, I’m not sure that you’re going to be able to repopulate file content using your model (on read/write/acquire events). At least, it’s not that simple. For one, straight forward implementation will certainly cause deadlocks with cache manager. File will be locked before dirty pages are flushed and your write will block. If you manage to get around this one, there is read-ahead issue colliding with cache manager quota. Imagine an aggressive read-ahead on your stub file(s) that consumes all CM threads. Now you?re writing file data (likely, cached, since you reuse file object opened by an app) on condition that CM?s dirty pages threshold is reached. This will block your write on CcCanIWrite, which will stay forever since all CM threads are consumed by read-aheads which are blocked by your writes blocked on CcCanIWrite. So, CM will not be able to free up its dirty pages, since there is no thread available for that.
So, I guess, you have to ask yourself, if this ?on demand? recall is worth the headaches.

4

The blocking of WriteFile is most likely on the paging IO resource. This can happen if the cache manager is doing read ahead. The paging read gets blocked by your driver and when you try to write to the file it blocks because the paging read is holding the paging IO resource and your write is trying to acquire it. Try setting FILE_RANDOM_ACCESS in the create options (before reissuing the IO) if the file is offline. This prevents the cache manager from issuing read aheads. It is possible and, in fact, the preferred method to recall on first read/write.