File system testing and filter related heisenbugs

I have an FSD and a test suite that exercises various aspects of it. One common failure in my tests has to do with deletion handling and I stronly suspect that a filter (or filters) is to blame.

A common test scenario goes something like this:

// first do
h = CreateFileW(L"foo", CREATE_NEW);
CloseHandle(h);
DeleteFileW(L"foo");

// then either
h = CreateFileW(L"foo", OPEN_EXISTING);
ASSERT(INVALID_HANDLE_VALUE == h);
ASSERT(ERROR_FILE_NOT_FOUND == GetLastError());
// fails with ERROR_ACCESS_DENIED (STATUS_DELETE_PENDING)

// or
h = CreateFileW(L"foo", CREATE_NEW);
ASSERT(INVALID_HANDLE_VALUE != h);
// fails with ERROR_FILE_EXISTS

It is clear then that the file persists (momentarily) after the call to DeleteFileW, which should not be possible according to the code above, but of course it is possible if some other system component has opened the “foo” file just before the IRP_MJ_SET_INFORMATION / FileDispositionInformation gets acted upon. My main suspect is a filter (or filters) that opens the file for its own purposes thus delaying the actual deletion of the file.

I am fairly certain that I have observed this behavior on NTFS as well, which reenforces my belief that this is not a real problem with my FSD but rather an aspect of how the Windows file system works (and perhaps a problem with my test suite). I am seeking the wisdom of this list: is this a common problem when testing your file systems? How do you deal with it? Is it possible to turn off the filter manager and see if the problem goes away?

[NOTE: While I could avoid using IoRegisterFileSystem when I create a FILE_DEVICE_DISK_FILE_SYSTEM file system, this is not possible when I use the Mup for my file systems.]

Bill Zissimopoulos
https://github.com/billziss-gh/winfsp

Because of a suspicion of mine here is an additional note for FSD gurus: my FSD does not implement oplocks yet.

Bill Zissimopoulos
https://github.com/billziss-gh/winfsp

Your failure is unlikely to be associated with oplocks, though it’s vaguely possible (oplocks could cause files to remain open for some short period of time).

Do you implement directory change notifications?

Disabiling filter manager is not a viable idea at this point. Are you running only stock filters?

Any application could trigger that behavior. That’s because deletion on Windows is an *intent* and it can be adjusted or changed by other applications and kernel mode filter driver.

Tony
OSR

Tony, thanks for your answer.

My FSD implements directory change notifications. I doubt though that a user mode application (other than my testsuite) somehow gets into it, because I do not add drives for the tested file systems. (Files are accessed using \?\GLOBALROOT\Device\Volume{GUID} or \{Server}{Share} syntax.)

I am running stock filters only (AFAIK). The problem happens more on Win10 rather than Win8 (where in fact it is quite rare).

Regarding my inquiry about oplocks (about which my knowledge is close to zero). I was under the impression that well-behaved filters use oplocks when opening files. Since my FSD does not implement them, I could be getting “heisenbugs” like the one described in the top post.

Bill

Actually, lots of OS components (in user mode) now use oplocks as a means of watching for directory changes. For example, the dynamic content on the Metro screen relies upon oplocks to monitor for changes and we’ve found that not implementing them can cause interesting problems.

One thing to do in your filter would be to watch for any other open/create activity in order to rule that out.

I also suspect that one complication for you is that deletion is *not* the same on Windows as it is on UNIX platforms. In Windows, deletion is an intent that is stored in the stream control block or the cache control block (using NTFS terminology here). For example, if you open a file with FILE_DELETE_ON_CLOSE the deletion intent is stored in the CCB and is “promoted” to the SCB when the file itself is closed (you can see this in the FAT source code). On the other hand, if you open a file for DELETE access and then set the disposition information (IRP_MJ_SET_INFORMATION) then the intent is stored in the SCB directly.

The intent in the SCB is reversible. The intent in the CCB is not reversible (prior to Redstone. There is a new set information operation in Windows 10 that changes this behavior). However, once the CCB status is promoted to the SCB, it can be reversed.

The deletion doesn’t happen until the file object gets closed. So if, for example, Mm has a reference against the file object, it won’t go away until Mm releases that reference. The file systems can do things to “hurry things along” but it’s a bit more difficult in a filter - you could try, for example, to send an IRP_MJ_FLUSH_BUFFERS with the IRP_MN_FLUSH_AND_PURGE option. FAT does not appear to implement this (at least the sample doesn’t do so). Clearly, the idea was to get filters to stop trying to call CcPurgeCache directly (which does not work right and filters that do so are broken). If implemented, this should encourage Mm to get rid of the cache more quickly.

Thus, you may just be observing the effect of lots of asynchronous behavior in the system. It could even be caused by something you do in your own filter (do you do any cached I/O operations?) We’ve certainly seen this sort of “things do not go away fast enough” issue in the past. One option is to retry, though you probably won’t want to wait for many seconds for this to finally work. Still, you might want to retry and confirm that it does eventually “go away”.

In a kernel debugger, you can see this by looking at the file object. FileObject->SectionObjectPointers will contain references to the Mm control areas (DataSectionObject and ImageSectionObject) and you can even use “!ca” on the address of those objects to find out more information about what memory manager activity is ongoing. You can also look at the shared cache map field (which is something like nt!_SHARED_CACHE_MAP as I recall) and you’ll see there is a file object potentially referenced in there as well.

I know these aren’t solutions, but hopefully these will help you understand what’s happening and why and ultimately figure out how to obtain the behavior that you want.

Tony
OSR

Tony, thanks for your elaborate answer.

To clarify my project is a full FSD and not a filter, but I get the gist of what you are saying.

I agree with you that DeleteFile is an intent and that in an asynchronous system like Windows actual deletion can be delayed arbitrarily. When I get some time I will investigate further and confirm whether it is a filter that is causing these issues.

Bill

That’s actually *much* easier then. So few people write FSDs these days - but in my experience they are much easier than anything other than trivial filter projects.

You need to make sure you are purging the cache of a delete pending file in the cleanup path. Just be careful, if you purge it in cleanup, it may succeed, which can trigger a close. It may also fail (memory mapped files).

Tony

> I have an FSD and a test suite that exercises various aspects of it. One

common failure in my tests has to do
with deletion handling and I stronly suspect that a filter (or filters) is
to blame.

Its not uncommon for AV filters to scan on cleanup.

I’m guessing that you already do the Cc[FlushAnd]Purge if the file is
deleted. In theory you then need to wait for the CcUninitialize to complete
(although FAT doesn’t and I don’t understand why), are you doing that
?(spoiler alert I’ve had mixed success with it)

Given that this is a (relatively) rare path, a CcWaitForLazyWriter (one of
the more terrifying weapons in the FSD developer’s arsenal) wouldn’t go
amiss - at least for testing purposes.

R

xxxxx@billz.fastmail.fm wrote:

I agree with you that DeleteFile is an intent and that in an asynchronous
system like Windows actual deletion can be delayed arbitrarily. When I get
some time I will investigate further and confirm whether it is a filter
that is causing these issues.

You might even start with just a simple Process Monitor log of whom is
doing what at the time of your STATUS_DELETE_PENDING observation.

The last one of those we chased down here was in our build process,
and it was the CSC (Windows Offline Files) service which had opened an
additional handle to a temporary file the build system was using. The
next component build expected to create & delete that same temporary
file, but could get STATUS_DELETE_PENDING because the CSC service
still had his file handle open.

Maybe that still fits with “a filter is doing this to me”, since
presumably a filter of file activity is what notified the CSC service
process to make this attempt. I’m just saying there was no subtlety
or magic to why STATUS_DELETE_PENDING occurred; even Process Monitor
showed that there was a user-mode handle still outstanding to the
deleted file when the next CreateFile occurred.

Alan Adams
Client for Open Enterprise Server
Micro Focus
xxxxx@microfocus.com

Thanks everyone for your suggestions. Some comments below.

Tony Mason wrote:

You need to make sure you are purging the cache of a delete pending file in the
cleanup path. Just be careful, if you purge it in cleanup, it may succeed,
which can trigger a close. It may also fail (memory mapped files).

Rod Widdowson wrote:

I’m guessing that you already do the Cc[FlushAnd]Purge if the file is
deleted. In theory you then need to wait for the CcUninitialize to complete
(although FAT doesn’t and I don’t understand why), are you doing that
?(spoiler alert I’ve had mixed success with it)

I am doing CcUnitializeCacheMap which my understanding is that it does flush and purge when the TruncateSize pointer is non-NULL and the SharedCacheMap is going away. My understanding may be wrong.

Thanks for the pointer regarding waiting for CcUnitialize to complete. I am currently not waiting and it may be worth trying to wait and see what happens.

Given that this is a (relatively) rare path, a CcWaitForLazyWriter (one of
the more terrifying weapons in the FSD developer’s arsenal) wouldn’t go
amiss - at least for testing purposes.

Another interesting pointer. I may try this for testing purposes.

–

I am currently in the process of finishing reparse point support, but hopefully will revisit this in the near future and report back my findings.

Bill

Are you supporting the new “reparse point on non-empty directory” support?

Tony
OSR

Tony Mason wrote:

Are you supporting the new “reparse point on non-empty directory” support?

I am not aware of any new “reparse point on non-empty directory” support requirement. I would love a link/pointer if you have it.

I am actually implementing an (open source) user mode file system solution called WinFsp:

https://github.com/billziss-gh/winfsp

It is up to the (user mode) file system whether to allow reparse points on non-empty directories. My test file system (memfs) does not:

https://github.com/billziss-gh/winfsp/blob/master/tst/memfs/memfs.cpp#L1012

Bill

They are in Redstone.

For example:

#if (NTDDI_VERSION >= NTDDI_WIN10_RS1)
NTKERNELAPI
BOOLEAN
FsRtlIsNonEmptyDirectoryReparsePointAllowed(
In ULONG ReparseTag
);
#endif

And a comment about this:

// The following flags control behavior when a reparse point is encountered
// on a directory that may be non-empty (one whose reparse tag is
// recognized by FsRtlIsNonEmptyDirectoryReparsePointAllowed):
//
// OPEN_REPARSE_POINT_REPARSE_IF_CHILD_EXISTS -
// If the reparse point is on a directory that is not the final path
// component and the next path component exists, reparse on the directory.
//
// OPEN_REPARSE_POINT_REPARSE_IF_CHILD_NOT_EXISTS -
// If the reparse point is on a directory that is not the final path
// component and the next path component does not exist, reparse on the
// directory.
//
// OPEN_REPARSE_POINT_REPARSE_IF_DIRECTORY_FINAL_COMPONENT -
// If the reparse point is on a directory that is the final path
// component, reparse on the directory unless FILE_OPEN_REPARSE_POINT
// is specified.
//
// Specifying all three of the above flags is legal and simply means always
// reparse on any directory reparse point.

There are a variety of other changes as well and I’m curious how people are handling them.

Tony

Interesting. The FsRtlIsNonEmptyDirectoryReparsePointAllowed API seems to return TRUE/FALSE based on the ReparseTag only. Perhaps there is a class of reparse points that are now allowed to be set on non-empty directories?

BTW, I am finding the comments you quoted on the documentation for the new OPEN_REPARSE_LIST_ENTRY:

https://msdn.microsoft.com/en-us/library/windows/hardware/mt734265(v=vs.85).aspx

So now one can supply an ECP with a list of reparse tags/guids that can be opened without getting STATUS_REPARSE. I am not sure what the relation to non-empty directories is though.

Bill

Bill/Tony:

So far only NTFS implements everything I’m about to describe. Some of this has been mentioned at PlugFest, and I’m sure will be mentioned again at the next PlugFest. I’ll be on hand at the next PlugFest.

Non-empty directory reparse points simply means that 1) you can now create a reparse point on a non-empty directory; and 2) if you have a directory with a reparse point you can now create files in that directory.

We didn’t want apply this change retroactively to existing reparse tags being used on directories, as that could break expectations. So we designed a new class of reparse point that opts in on a tag by tag basis. It is essentially based on this new reparse tag flag:

// ±±±±±----------------------±------------------------------+
// |M|R|N|D| Reserved bits | Reparse Tag Value |
// ±±±±±----------------------±------------------------------+
//
// D is the directory bit. When set to 1, indicates that any directory
// with this reparse tag can have children. Has no special meaning when used
// on a non-directory file. Not compatible with the name surrogate bit.
//

//
// Macro to determine whether a directory with this reparse point can have
// children.
//

#define IsReparseTagDirectory(_tag) ( \
((_tag) & 0x10000000) \
)

However, it doesn’t map perfectly to that flag, because we had to grandfather in an existing reparse point tag that needs this behavior, and it’s possible we’ll do future tweaks to include/exclude certain tags. Hence we created an actual routine FsRtlIsNonEmptyDirectoryReparsePointAllowed to abstract the logic. Please call this routine and do not rely on the bit.

Incidentally if third parties want this new behavior, they can ask for it when requesting new reparse point tags.

Non-empty directory reparse points are related in a couple of ways to OPEN_REPARSE_ECP. 1) The OPEN_REPARSE_POINT_REPARSE_IF_CHILD_EXISTS, OPEN_REPARSE_POINT_REPARSE_IF_CHILD_NOT_EXISTS, and OPEN_REPARSE_POINT_REPARSE_IF_DIRECTORY_FINAL_COMPONENT flags only apply to this class of reparse points. 2) If you fill in an OPEN_REPARSE_ECP, you have full control over when/if NTFS reparses or not. But when we thought about “offline” scenarios (dual-boot, filter got uninstalled, etc.) we wanted these directories to act seamlessly, since they are in essence directories. You should be able to cd into them, dir them, etc. without having problems. So we put special behavior in NTFS, guided by FsRtlIsNonEmptyDirectoryReparsePointAllowed, when an OPEN_REPARSE_ECP is NOT present. I’m going to be lazy and just paste a comment from NTFS which describes this pretty well.

//
// Our decision whether to reparse or not depends on various factors such
// as what type of reparse tag we encountered, whether an OPEN_REPARSE
// ECP is present, whether this is a file or directory, etc.
//
//
// “OFFLINE” BEHAVIOR (OpenReparseTag == NULL):
// ------------------
//
// When a reparse point is encountered that is NOT a directory that may
// be non-empty (one whose reparse tag is recognized by
// FsRtlIsNonEmptyDirectoryReparsePointAllowed), then reparse. This is
// traditional file system behavior.
//
// The following flags control behavior when a reparse point is encountered
// that IS on a directory that may be non-empty (one whose reparse tag is
// recognized by FsRtlIsNonEmptyDirectoryReparsePointAllowed), then do not
// reparse. This is custom behavior for this class of reparse points.
//
//
// “ONLINE” BEHAVIOR (OpenReparseTag != NULL):
// -----------------
//
// When a reparse point is encountered that is NOT a directory that may
// be non-empty (one whose reparse tag is recognized by
// FsRtlIsNonEmptyDirectoryReparsePointAllowed), then do not reparse.
//
// The following flags control behavior when a reparse point is encountered
// that IS on a directory that may be non-empty (one whose reparse tag is
// recognized by FsRtlIsNonEmptyDirectoryReparsePointAllowed):
//
// OPEN_REPARSE_POINT_REPARSE_IF_CHILD_EXISTS -
// If the reparse point is on a directory that is not the final path
// component and the next path component exists, reparse on the directory.
//
// OPEN_REPARSE_POINT_REPARSE_IF_CHILD_NOT_EXISTS -
// If the reparse point is on a directory that is not the final path
// component and the next path component does not exist, reparse on the
// directory.
//
// OPEN_REPARSE_POINT_REPARSE_IF_DIRECTORY_FINAL_COMPONENT -
// If the reparse point is on a directory that is the final path
// component, reparse on the directory unless FILE_OPEN_REPARSE_POINT
// is specified.
//
// Specifying all three of the above flags is legal and simply means always
// reparse on any directory reparse point.
//

The main challenges implementing this in your own file system are structuring the code so that you can “peek” ahead into a directory to see if the next component exists, and also getting all of the flag combinations correct.

Craig (MSFT)

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@billz.fastmail.fm
Sent: Monday, September 19, 2016 6:00 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] File system testing and filter related heisenbugs

Interesting. The FsRtlIsNonEmptyDirectoryReparsePointAllowed API seems to return TRUE/FALSE based on the ReparseTag only. Perhaps there is a class of reparse points that are now allowed to be set on non-empty directories?

BTW, I am finding the comments you quoted on the documentation for the new OPEN_REPARSE_LIST_ENTRY:

https://msdn.microsoft.com/en-us/library/windows/hardware/mt734265(v=vs.85).aspx

So now one can supply an ECP with a list of reparse tags/guids that can be opened without getting STATUS_REPARSE. I am not sure what the relation to non-empty directories is though.

Bill

—
NTFSD is sponsored by OSR

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:>

This is fantastic information. Thank you, Craig.