AcquireForCcFlush callback and lazy writes on files with FO_CLEANUP_COMPLETE set

NTFSD
1

I have a simple minifilter which just monitors various file operations. It registers for pretty much all the callbacks including acquire/release for CcFlush. I do track certain states in the stream context.

I have observed that AcquireforCCFlush callback is not called for files on which last cleanup has happened and before the lazy writer flushes the data. AcquireForCcFlush is working properly for non cleanedup files.

I see paging IO writes on files with FO_CLEANUP_COMPLETE set and ref counts of 2 (!object fileobj). Following is the call stack I see… Top level IRP is FSRTL_CACHE_TOP_LEVEL_IRP.

f9c1a64c 805293ee 0000001a 8196f148 81968494 nt!DbgBreakPoint (FPO: [0,0,0])
f9c1a93c 80529430 f9a65a80 f9a65fe0 00000096 nt!RtlAssert2+0x104 (FPO: [5,183,4])
f9c1a958 f9a64608 f9a65a80 f9a65fe0 00000096 nt!RtlAssert+0x18 (FPO: [4,0,0])
f9c1a9f8 f9a649c1 81968494 f9c1aa58 f9c1aa88 foobarFilter!foobarFltCommonWrite+0x108 (FPO: [Non-Fpo])
f9c1aa0c bae4516b 81968494 f9c1aa58 f9c1aa88 foobarFilter!foobarFltPreOpWrite+0x61 (FPO: [Non-Fpo])
f9c1aa38 bae2c888 00000001 00000004 f9c1aa88 fltmgr!FltvPreOperation+0x3f (FPO: [3,3,4])
f9c1aa98 bae2e2a0 00c1aae0 00000000 f9c1aae0 fltmgr!FltpPerformPreCallbacks+0x2d4 (FPO: [1,16,0])
f9c1aaac bae2ec48 f9c1aae0 00000000 8190eee8 fltmgr!FltpPassThroughInternal+0x32 (FPO: [2,0,0])
f9c1aac8 bae2f059 f9c1aa00 8190ee00 8196f040 fltmgr!FltpPassThrough+0x1c2 (FPO: [1,0,4])
f9c1aaf8 804ee119 8190eee8 81498940 806d22e8 fltmgr!FltpDispatch+0x10d (FPO: [2,6,0])
f9c1ab08 8064d628 817030c0 00025000 8190eee8 nt!IopfCallDriver+0x31 (FPO: [0,0,0])
f9c1ab2c 804ef411 f9c1ab68 f9c1ad40 00000000 nt!IovCallDriver+0xa0 (FPO: [0,4,0])
f9c1ab40 8050c497 8170300c f9c1ab68 f9c1abfc nt!IoSynchronousPageWrite+0xaf (FPO: [5,0,4])
f9c1ac24 8050ce3d e182a130 e182a138 e182a138 nt!MiFlushSectionInternal+0x3bf (FPO: [6,47,4])
f9c1ac60 804e38a2 8173a590 e182a130 00000025 nt!MmFlushSection+0x1b5 (FPO: [5,4,0])
f9c1ace8 804e3bc4 00001000 00000000 00000001 nt!CcFlushCache+0x386 (FPO: [4,24,4])
f9c1ad2c 804e61ee 819d5d50 8055b0c0 819a0530 nt!CcWriteBehind+0xdc (FPO: [0,8,4])
f9c1ad74 80534c02 819d5d50 00000000 819a0530 nt!CcWorkerThread+0x126 (FPO: [Non-Fpo])
f9c1adac 805c6160 819d5d50 00000000 00000000 nt!ExpWorkerThread+0x100 (FPO: [1,8,0])
f9c1addc 80541dd2 80534b02 00000000 00000000 nt!PspSystemThreadStartup+0x34 (FPO: [Non-Fpo])

I would like to understand why is AccquireforCCFlush not invoked? I can see FCB->PagingIoResource being already taken by the same thread which ensure correct synchronization between fsd and ntfs. Does CC acquire this resource explicitly bypassing filters?

2

The cache manager’s acquire for lazywrite/read ahead callbacks are not filterable. The AcquireForCcFlush callback you see only happens when a file is explicitly flushed (or Mm explicitly performs the flush as part of some cleanup work) - not when it is lazily flushed back to disk. File systems provide callbacks to the cache manager and it calls them directly.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Tuesday, May 26, 2009 6:21 PM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] AcquireForCcFlush callback and lazy writes on files with FO_CLEANUP_COMPLETE set

I have a simple minifilter which just monitors various file operations. It registers for pretty much all the callbacks including acquire/release for CcFlush. I do track certain states in the stream context.

I have observed that AcquireforCCFlush callback is not called for files on which last cleanup has happened and before the lazy writer flushes the data. AcquireForCcFlush is working properly for non cleanedup files.

I see paging IO writes on files with FO_CLEANUP_COMPLETE set and ref counts of 2 (!object fileobj). Following is the call stack I see… Top level IRP is FSRTL_CACHE_TOP_LEVEL_IRP.

f9c1a64c 805293ee 0000001a 8196f148 81968494 nt!DbgBreakPoint (FPO: [0,0,0])
f9c1a93c 80529430 f9a65a80 f9a65fe0 00000096 nt!RtlAssert2+0x104 (FPO: [5,183,4])
f9c1a958 f9a64608 f9a65a80 f9a65fe0 00000096 nt!RtlAssert+0x18 (FPO: [4,0,0])
f9c1a9f8 f9a649c1 81968494 f9c1aa58 f9c1aa88 foobarFilter!foobarFltCommonWrite+0x108 (FPO: [Non-Fpo])
f9c1aa0c bae4516b 81968494 f9c1aa58 f9c1aa88 foobarFilter!foobarFltPreOpWrite+0x61 (FPO: [Non-Fpo])
f9c1aa38 bae2c888 00000001 00000004 f9c1aa88 fltmgr!FltvPreOperation+0x3f (FPO: [3,3,4])
f9c1aa98 bae2e2a0 00c1aae0 00000000 f9c1aae0 fltmgr!FltpPerformPreCallbacks+0x2d4 (FPO: [1,16,0])
f9c1aaac bae2ec48 f9c1aae0 00000000 8190eee8 fltmgr!FltpPassThroughInternal+0x32 (FPO: [2,0,0])
f9c1aac8 bae2f059 f9c1aa00 8190ee00 8196f040 fltmgr!FltpPassThrough+0x1c2 (FPO: [1,0,4])
f9c1aaf8 804ee119 8190eee8 81498940 806d22e8 fltmgr!FltpDispatch+0x10d (FPO: [2,6,0])
f9c1ab08 8064d628 817030c0 00025000 8190eee8 nt!IopfCallDriver+0x31 (FPO: [0,0,0])
f9c1ab2c 804ef411 f9c1ab68 f9c1ad40 00000000 nt!IovCallDriver+0xa0 (FPO: [0,4,0])
f9c1ab40 8050c497 8170300c f9c1ab68 f9c1abfc nt!IoSynchronousPageWrite+0xaf (FPO: [5,0,4])
f9c1ac24 8050ce3d e182a130 e182a138 e182a138 nt!MiFlushSectionInternal+0x3bf (FPO: [6,47,4])
f9c1ac60 804e38a2 8173a590 e182a130 00000025 nt!MmFlushSection+0x1b5 (FPO: [5,4,0])
f9c1ace8 804e3bc4 00001000 00000000 00000001 nt!CcFlushCache+0x386 (FPO: [4,24,4])
f9c1ad2c 804e61ee 819d5d50 8055b0c0 819a0530 nt!CcWriteBehind+0xdc (FPO: [0,8,4])
f9c1ad74 80534c02 819d5d50 00000000 819a0530 nt!CcWorkerThread+0x126 (FPO: [Non-Fpo])
f9c1adac 805c6160 819d5d50 00000000 00000000 nt!ExpWorkerThread+0x100 (FPO: [1,8,0])
f9c1addc 80541dd2 80534b02 00000000 00000000 nt!PspSystemThreadStartup+0x34 (FPO: [Non-Fpo])

I would like to understand why is AccquireforCCFlush not invoked? I can see FCB->PagingIoResource being already taken by the same thread which ensure correct synchronization between fsd and ntfs. Does CC acquire this resource explicitly bypassing filters?

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

3

Thanks Apurva for your comments. I guess this forces me to go back to the drawing board. In my filter I need to ensure that while there are any outstanding IO (paging/non-paging) operations on a file, the file should not be extended/truncated and vice versa. I need to implement it at the fs filter layer and don?t want to rely on fsd. I am considering using a simple resource which I acquire shared during IO operations and ex during setinfo. To ensure lock ordering (myfilter first then fsd because that?s whats gonna happen in setinfo) I have registered callbacks for ccflush/mod write etc… But as you said CC lazy writer/read ahead threads don?t call into filters, I wont be able to ensure this lock ordering…

What would you suggest, how should I go about accomplishing this…

I am digging through the old archives incase this topic has already been discussed. If you know of any please do share.

Thanks
-Kiran

4

I think you are going to find that holding a lock while performing IO is
a bad idea, and can lead to deadlocks (due to reentrancy, among other
things). It should be fairly easy to corroborate this with information
from the archives.

At this point, the standard question arises: Why are you trying to
prevent extension and truncation while there is outstanding IO on the
file?

~Eric

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: Wednesday, May 27, 2009 3:46 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] AcquireForCcFlush callback and lazy
writes on files with FO_CLEANUP_COMPLETE set

Thanks Apurva for your comments. I guess this forces me to go
back to the drawing board. In my filter I need to ensure that
while there are any outstanding IO (paging/non-paging)
operations on a file, the file should not be
extended/truncated and vice versa. I need to implement it at
the fs filter layer and don?t want to rely on fsd. I am
considering using a simple resource which I acquire shared
during IO operations and ex during setinfo. To ensure lock
ordering (myfilter first then fsd because that?s whats gonna
happen in setinfo) I have registered callbacks for
ccflush/mod write etc… But as you said CC lazy writer/read
ahead threads don?t call into filters, I wont be able to
ensure this lock ordering…

What would you suggest, how should I go about accomplishing this…

I am digging through the old archives incase this topic has
already been discussed. If you know of any please do share.

Thanks
-Kiran

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online
at http://www.osronline.com/page.cfm?name=ListServer

5

Eric,

I did consider the reentrancy issues to some length and was hoping could be addressed by using two locks (main/paging). After a bit careful examination it seems that approach is not going to fly either. Irrespective of reentrancy issues I guess my model is broken anyways as the lazy writer/read ahead threads dont go through the filter stack for lock acquisitions and it seems there is no clean way I will be able to guarantee lock ordering I am looking for: myfilter->fsd->cc->mm

Why? : I want to ensure that the flile’s layout does not change while there are outstanding IOs. File extensions are not a big problem for us but truncates are.

thanks
-Kiran

6

The issue is that CcInitializeCacheMap registers callback functions.
This is a registration API, not an IRP API and thus it is not something
you can filter.

The Cache Manager calls these functions directly. Your filter does not
see them.

If you want to see these, you must be the file system; for a “filter”
this is what we usually refer to as a layered file system. In UNIX
systems the term stackable file system is generally used. Mini-filters
can also use the “shadow file object” technique, which layers them. You
then must implement the cache management logic yourself - you gain
tremendous control at the cost of implementation complexity.

And thus you learn a lesson I’ve commented on in this forum more than
once: building filters is often more complicated than building file
systems.

Tony
OSR

7

>I need to ensure that while there are any outstanding IO (paging/non-paging) operations on a file, the

file should not be extended/truncated and vice versa.

Why is this requirement?

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

8

> Why? : I want to ensure that the flile’s layout does not change while there are outstanding IOs. File

extensions are not a big problem for us but truncates are.

This is usually safe “by itself”, since the FSD’s file truncation will be stalled till there will be no active IO on the file.

For instance, FASTFAT uses ->PagingIoResource only for this purpose.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com