Detecting file deletion in minifilter

I want to track file deletions through my minifilter. I tracked file delete with FileDispositionInformation and IRP_MJ_SET_INFORMATION. But could not track FILE_DELETE_ON_CLOSE delete operation. Test code which opens an exising file. Code snippet is as below

InitializeObjectAttributes( &objAttributes,&originFile,
OBJ_CASE_INSENSITIVE | OBJ_KERNEL_HANDLE,NULL,NULL );

status = ZwCreateFile(&FileHandle,
FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES,
&objAttributes, &IoStatusBlock, NULL,
FILE_ATTRIBUTE_NORMAL, FILE_SHARE_DELETE,FILE_OPEN,
FILE_NON_DIRECTORY_FILE | FILE_DELETE_ON_CLOSE, NULL,0
);

ASSERT(STATUS_SUCCESS == status);

ZwClose(FileHandle);

But neither DeletePending was set nor any related flag was set for that file object in pre create, post create, pre cleanup and post cleanup callbacks

Am I missing something ?

I referred https://www.osronline.com/showthread.cfm?link=133727, http://www.osronline.com/showThread.cfm?link=16430, http://www.osronline.com/showThread.cfm?link=67122. But still not clear.

When you open a file with FILE_DELETE_ON_CLOSE, this flag will be present in
Data->Iopb->Parameters.Create.Options.

Now you have to create a file context (fltmgr supports file context from
Vista+ so you have to create your own implementation) which will indicate
delete pending status. This flag will be set to TRUE if the file is opened
with FILE_DELETE_ON_CLOSE flag. Note, you have to check if the file is not
hardlink, then you *may* ignore the file, because it may not be from your
monitored folder where you track file deletions. If IRP_MJ_SET_INFORMATION
is called with FileDispositionInformation/DeleteFile = FALSE, then set the
flag to FALSE (!!! ONLY if the file was not opened before with
FILE_DELETE_ON_CLOSE, because this delete status cannot be unchecked - find
more details about this in archive). In IRP_MJ_CLOSE check if the flag is
still TRUE.

Petr Kurtin

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: 6. listopadu 2008 11:02
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Detecting file deletion in minifilter

I want to track file deletions through my minifilter. I tracked file delete
with FileDispositionInformation and IRP_MJ_SET_INFORMATION. But could not
track FILE_DELETE_ON_CLOSE delete operation. Test code which opens an
exising file. Code snippet is as below

InitializeObjectAttributes( &objAttributes,&originFile,
OBJ_CASE_INSENSITIVE | OBJ_KERNEL_HANDLE,NULL,NULL );

status = ZwCreateFile(&FileHandle,
FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES,
&objAttributes, &IoStatusBlock, NULL,
FILE_ATTRIBUTE_NORMAL,
FILE_SHARE_DELETE,FILE_OPEN,
FILE_NON_DIRECTORY_FILE |
FILE_DELETE_ON_CLOSE, NULL,0
);

ASSERT(STATUS_SUCCESS == status);

ZwClose(FileHandle);

But neither DeletePending was set nor any related flag was set for that file
object in pre create, post create, pre cleanup and post cleanup callbacks

Am I missing something ?

I referred https://www.osronline.com/showthread.cfm?link=133727,
http://www.osronline.com/showThread.cfm?link=16430,
http://www.osronline.com/showThread.cfm?link=67122. But still not clear.


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@avast.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You could store the flag in a Stream Handle Context, which is supported
in 2000 SP4 UR1, XP, Server 2003 and later.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Petr Kurtin
Sent: 06 November 2008 13:22
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Detecting file deletion in minifilter

*** WARNING ***

This mail has originated outside your organization, either from an
external partner or the Global Internet.
Keep this in mind if you answer this message.

When you open a file with FILE_DELETE_ON_CLOSE, this flag will be
present in
Data->Iopb->Parameters.Create.Options.

Now you have to create a file context (fltmgr supports file context from
Vista+ so you have to create your own implementation) which will
Vista+ indicate
delete pending status. This flag will be set to TRUE if the file is
opened with FILE_DELETE_ON_CLOSE flag. Note, you have to check if the
file is not hardlink, then you *may* ignore the file, because it may not
be from your monitored folder where you track file deletions. If
IRP_MJ_SET_INFORMATION is called with
FileDispositionInformation/DeleteFile = FALSE, then set the flag to
FALSE (!!! ONLY if the file was not opened before with
FILE_DELETE_ON_CLOSE, because this delete status cannot be unchecked -
find more details about this in archive). In IRP_MJ_CLOSE check if the
flag is still TRUE.

Petr Kurtin

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: 6. listopadu 2008 11:02
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Detecting file deletion in minifilter

I want to track file deletions through my minifilter. I tracked file
delete with FileDispositionInformation and IRP_MJ_SET_INFORMATION. But
could not track FILE_DELETE_ON_CLOSE delete operation. Test code which
opens an exising file. Code snippet is as below

InitializeObjectAttributes( &objAttributes,&originFile,
OBJ_CASE_INSENSITIVE | OBJ_KERNEL_HANDLE,NULL,NULL );

status = ZwCreateFile(&FileHandle,
FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES,
&objAttributes, &IoStatusBlock, NULL,
FILE_ATTRIBUTE_NORMAL,
FILE_SHARE_DELETE,FILE_OPEN,
FILE_NON_DIRECTORY_FILE |
FILE_DELETE_ON_CLOSE, NULL,0
);

ASSERT(STATUS_SUCCESS == status);

ZwClose(FileHandle);

But neither DeletePending was set nor any related flag was set for that
file object in pre create, post create, pre cleanup and post cleanup
callbacks

Am I missing something ?

I referred https://www.osronline.com/showthread.cfm?link=133727,
http://www.osronline.com/showThread.cfm?link=16430,
http://www.osronline.com/showThread.cfm?link=67122. But still not clear.


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@avast.com To
unsubscribe send a blank email to xxxxx@lists.osr.com


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@baesystems.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************

I think it’s not possible to use stream handle contexts, because
FILE_DELETE_ON_CLOSE is associated with all opens accross all data streams
for a given file. Is it right?

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Oliver, Jonathan
(UK)
Sent: 6. listopadu 2008 15:19
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Detecting file deletion in minifilter

You could store the flag in a Stream Handle Context, which is supported
in 2000 SP4 UR1, XP, Server 2003 and later.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Petr Kurtin
Sent: 06 November 2008 13:22
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Detecting file deletion in minifilter

*** WARNING ***

This mail has originated outside your organization, either from an
external partner or the Global Internet.
Keep this in mind if you answer this message.

When you open a file with FILE_DELETE_ON_CLOSE, this flag will be
present in
Data->Iopb->Parameters.Create.Options.

Now you have to create a file context (fltmgr supports file context from
Vista+ so you have to create your own implementation) which will
Vista+ indicate
delete pending status. This flag will be set to TRUE if the file is
opened with FILE_DELETE_ON_CLOSE flag. Note, you have to check if the
file is not hardlink, then you *may* ignore the file, because it may not
be from your monitored folder where you track file deletions. If
IRP_MJ_SET_INFORMATION is called with
FileDispositionInformation/DeleteFile = FALSE, then set the flag to
FALSE (!!! ONLY if the file was not opened before with
FILE_DELETE_ON_CLOSE, because this delete status cannot be unchecked -
find more details about this in archive). In IRP_MJ_CLOSE check if the
flag is still TRUE.

Petr Kurtin

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: 6. listopadu 2008 11:02
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Detecting file deletion in minifilter

I want to track file deletions through my minifilter. I tracked file
delete with FileDispositionInformation and IRP_MJ_SET_INFORMATION. But
could not track FILE_DELETE_ON_CLOSE delete operation. Test code which
opens an exising file. Code snippet is as below

InitializeObjectAttributes( &objAttributes,&originFile,
OBJ_CASE_INSENSITIVE | OBJ_KERNEL_HANDLE,NULL,NULL );

status = ZwCreateFile(&FileHandle,
FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES,
&objAttributes, &IoStatusBlock, NULL,
FILE_ATTRIBUTE_NORMAL,
FILE_SHARE_DELETE,FILE_OPEN,
FILE_NON_DIRECTORY_FILE |
FILE_DELETE_ON_CLOSE, NULL,0
);

ASSERT(STATUS_SUCCESS == status);

ZwClose(FileHandle);

But neither DeletePending was set nor any related flag was set for that
file object in pre create, post create, pre cleanup and post cleanup
callbacks

Am I missing something ?

I referred https://www.osronline.com/showthread.cfm?link=133727,
http://www.osronline.com/showThread.cfm?link=16430,
http://www.osronline.com/showThread.cfm?link=67122. But still not clear.


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@avast.com To
unsubscribe send a blank email to xxxxx@lists.osr.com


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@baesystems.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

I think its possible to delete individual streams from a file without
deleting the entire file, in which case FILE_DELETE_ON_CLOSE would be
associated with the stream of the file object being opened for deletion?

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Petr Kurtin
Sent: 06 November 2008 14:49
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Detecting file deletion in minifilter

I think it’s not possible to use stream handle contexts, because
FILE_DELETE_ON_CLOSE is associated with all opens accross all data
streams for a given file. Is it right?

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Oliver, Jonathan
(UK)
Sent: 6. listopadu 2008 15:19
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Detecting file deletion in minifilter

You could store the flag in a Stream Handle Context, which is supported
in 2000 SP4 UR1, XP, Server 2003 and later.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Petr Kurtin
Sent: 06 November 2008 13:22
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Detecting file deletion in minifilter

*** WARNING ***

This mail has originated outside your organization, either from an
external partner or the Global Internet.
Keep this in mind if you answer this message.

When you open a file with FILE_DELETE_ON_CLOSE, this flag will be
present in
Data->Iopb->Parameters.Create.Options.

Now you have to create a file context (fltmgr supports file context from
Vista+ so you have to create your own implementation) which will
Vista+ indicate
delete pending status. This flag will be set to TRUE if the file is
opened with FILE_DELETE_ON_CLOSE flag. Note, you have to check if the
file is not hardlink, then you *may* ignore the file, because it may not
be from your monitored folder where you track file deletions. If
IRP_MJ_SET_INFORMATION is called with
FileDispositionInformation/DeleteFile = FALSE, then set the flag to
FALSE (!!! ONLY if the file was not opened before with
FILE_DELETE_ON_CLOSE, because this delete status cannot be unchecked -
find more details about this in archive). In IRP_MJ_CLOSE check if the
flag is still TRUE.

Petr Kurtin

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: 6. listopadu 2008 11:02
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Detecting file deletion in minifilter

I want to track file deletions through my minifilter. I tracked file
delete with FileDispositionInformation and IRP_MJ_SET_INFORMATION. But
could not track FILE_DELETE_ON_CLOSE delete operation. Test code which
opens an exising file. Code snippet is as below

InitializeObjectAttributes( &objAttributes,&originFile,
OBJ_CASE_INSENSITIVE | OBJ_KERNEL_HANDLE,NULL,NULL );

status = ZwCreateFile(&FileHandle,
FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES,
&objAttributes, &IoStatusBlock, NULL,
FILE_ATTRIBUTE_NORMAL,
FILE_SHARE_DELETE,FILE_OPEN,
FILE_NON_DIRECTORY_FILE |
FILE_DELETE_ON_CLOSE, NULL,0
);

ASSERT(STATUS_SUCCESS == status);

ZwClose(FileHandle);

But neither DeletePending was set nor any related flag was set for that
file object in pre create, post create, pre cleanup and post cleanup
callbacks

Am I missing something ?

I referred https://www.osronline.com/showthread.cfm?link=133727,
http://www.osronline.com/showThread.cfm?link=16430,
http://www.osronline.com/showThread.cfm?link=67122. But still not clear.


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@avast.com To
unsubscribe send a blank email to xxxxx@lists.osr.com


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@baesystems.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@baesystems.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

You can delete individual streams from a file with FILE_DELETE_ON_CLOSE and the other streams will be left alone except if you try to delete the ::$DATA stream. In this case the whole file will be deleted.

Regards,
Alex.
This posting is provided “AS IS” with no warranties, and confers no rights.

Detecting file deletion is a horrible mess from a mini-filter and,
ultimately, cannot be 100% achieved correctly.

Let’s look at FAT first (ergo, let’s pretend streams don’t exist.) If
you open the file with FILE_DELETE_ON_CLOSE the attribute to delete the
file is set in the CCB.

{
PCCB Ccb;
Ccb = (PCCB)FileObject->FsContext2;

//
// Mark the DeleteOnClose bit if the operation was
successful.
//

if ( DeleteOnClose ) {

SetFlag( Ccb->Flags, CCB_FLAG_DELETE_ON_CLOSE );
}

If you open the file for DELETE access and then set (or clear) the
disposition, the delete the file bit is set (or cleared) in the FCB:

//
// At this point either we have a file or an empty directory
// so we know the delete can proceed.
//

SetFlag( Fcb->FcbState, FCB_STATE_DELETE_ON_CLOSE );
FileObject->DeletePending = TRUE;

The problem with this behavior is that it creates a situation in which
the filter can never know if the file is really being deleted, since it
can be “undeleted” right up to the last moment (e.g., it could be
“undeleted” by a filter below you even.) However, this implementation
is peculiar in that it means if you open it “DELETE_ON_CLOSE” an
application can never directly reverse that decision (because set/clear
disposition impacts the FCB and not the CCB.) The CCB bit is passed to
the FCB on cleanup:

//
// Do a check here if this was a DELETE_ON_CLOSE FileObject,
and
// set the Fcb flag appropriately.
//

if (FlagOn(Ccb->Flags, CCB_FLAG_DELETE_ON_CLOSE)) {

ASSERT( NodeType(Fcb) != FAT_NTC_ROOT_DCB );

SetFlag(Fcb->FcbState, FCB_STATE_DELETE_ON_CLOSE);

Of course, this leads to really crazy scenarios: an application opens
the file (normally) and specifies shared delete semantics. It then
opens the file (delete on close) in a consistent sharing manner.
Depending upon whether or not the first or second handle gets closed,
you may or may not see the file “stop being deleted”.

I logged a bug against this about five years ago for several reasons:
(1) it’s not the same as what happens in NTFS; (2) it’s almost
impossible to explain to people why they see erratic behavior based upon
“order in which the handle gets closed”; (3) it makes detecting deletion
(and handling it) in a filter driver even more challenging (well,
probably not, but at first blush it looks that way.)

No doubt to spite me, the NTFS semantics (which were rational) have been
changed in recent releases to match the FAT semantics (so we now have
two irrational, “depends on the order of close” and “screw you filter
drivers” approaches to the world.)

Prior to the “let’s make NTFS behave more erratically and
inconsistently” change, NTFS used the FCB for the “delete on close”
option so an application could “change its mind” before the handle was
closed in either case. Thus, it had a consistent and understandable
behavior pattern…

Ah, but I started off by saying “let’s ignore streams.” Now let’s stop
ignoring them.

If I open a file without a stream specified (on NTFS, your mileage may
vary with UDFS and RDR, both of which also support streams,) I am
opening the “default data stream.” It’s the same as specifying
“::$DATA” to the end of the file name. It has special behavior. As a
filter driver, though, I’m watching stream contexts (Vista does give you
file contexts and you can build them yourself - sort of - in prior
versions.) So here’s the weird part: opening the default data stream in
specific modes may delete the alternate data streams.

Here’s a specific example: if I open the “:Zone_Identifier:$DATA” stream
normally, and then I open the “::$DATA” stream with Overwrite
disposition, when I close the zone identifier stream, it will be
deleted. If I instead opened with supersede, the open would fail.

Thus, the stream could be deleted without anyone ever explicitly
deleting it…

But maybe you don’t care about streams being deleted. Maybe you only
care about files being deleted. I’m not sure what the behavior of
various dispositions would be when combined with ADS access, but I’ve
always assumed they will defy comprehension.

What does this all mean? It means that you cannot reliably know when
files (or streams) are deleted. A filter below you could change the
behavior. The order in which handles get closed may mean that one time
the file has been undeleted and another time it has not. If you query
on IRP_MJ_CLEANUP to see if it is delete pending, there is no guarantee
that there aren’t other opens on the file that won’t reverse that
decision (note: it doesn’t require another filter below you - even an
application can do this. Anyone building test suites that doesn’t test
this scenario is slacking.)

So every time I look at this I conclude: “cannot 100% reliably detect a
delete from a filter.” The suggestion always arises then that “well,
just look in the directory after we close the file and see if it really
went away.” Even a modicum of analysis will make you realize that this
just won’t work reliably (ergo, someone else races in and creates the
file while you’re checking in the directory.) One of the disadvantages
of living in a highly parallel environment.

Maybe “most of the time” is good enough for your filter (“it doesn’t
crash very often!”) I find getting the cases where I know it will work
correct, that I shy away from the cases where I know it cannot always
work.

Tony
OSR

Hi Tony

Looked at the behaviour of DeleteFileTransacted have you? Expect more
pleasant surprises!

Best Wishes
Lyndon

“Tony Mason” wrote in message news:xxxxx@ntfsd…
Detecting file deletion is a horrible mess from a mini-filter and,
ultimately, cannot be 100% achieved correctly.

Let’s look at FAT first (ergo, let’s pretend streams don’t exist.) If
you open the file with FILE_DELETE_ON_CLOSE the attribute to delete the
file is set in the CCB.

{
PCCB Ccb;
Ccb = (PCCB)FileObject->FsContext2;

//
// Mark the DeleteOnClose bit if the operation was
successful.
//

if ( DeleteOnClose ) {

SetFlag( Ccb->Flags, CCB_FLAG_DELETE_ON_CLOSE );
}

If you open the file for DELETE access and then set (or clear) the
disposition, the delete the file bit is set (or cleared) in the FCB:

//
// At this point either we have a file or an empty directory
// so we know the delete can proceed.
//

SetFlag( Fcb->FcbState, FCB_STATE_DELETE_ON_CLOSE );
FileObject->DeletePending = TRUE;

The problem with this behavior is that it creates a situation in which
the filter can never know if the file is really being deleted, since it
can be “undeleted” right up to the last moment (e.g., it could be
“undeleted” by a filter below you even.) However, this implementation
is peculiar in that it means if you open it “DELETE_ON_CLOSE” an
application can never directly reverse that decision (because set/clear
disposition impacts the FCB and not the CCB.) The CCB bit is passed to
the FCB on cleanup:

//
// Do a check here if this was a DELETE_ON_CLOSE FileObject,
and
// set the Fcb flag appropriately.
//

if (FlagOn(Ccb->Flags, CCB_FLAG_DELETE_ON_CLOSE)) {

ASSERT( NodeType(Fcb) != FAT_NTC_ROOT_DCB );

SetFlag(Fcb->FcbState, FCB_STATE_DELETE_ON_CLOSE);

Of course, this leads to really crazy scenarios: an application opens
the file (normally) and specifies shared delete semantics. It then
opens the file (delete on close) in a consistent sharing manner.
Depending upon whether or not the first or second handle gets closed,
you may or may not see the file “stop being deleted”.

I logged a bug against this about five years ago for several reasons:
(1) it’s not the same as what happens in NTFS; (2) it’s almost
impossible to explain to people why they see erratic behavior based upon
“order in which the handle gets closed”; (3) it makes detecting deletion
(and handling it) in a filter driver even more challenging (well,
probably not, but at first blush it looks that way.)

No doubt to spite me, the NTFS semantics (which were rational) have been
changed in recent releases to match the FAT semantics (so we now have
two irrational, “depends on the order of close” and “screw you filter
drivers” approaches to the world.)

Prior to the “let’s make NTFS behave more erratically and
inconsistently” change, NTFS used the FCB for the “delete on close”
option so an application could “change its mind” before the handle was
closed in either case. Thus, it had a consistent and understandable
behavior pattern…

Ah, but I started off by saying “let’s ignore streams.” Now let’s stop
ignoring them.

If I open a file without a stream specified (on NTFS, your mileage may
vary with UDFS and RDR, both of which also support streams,) I am
opening the “default data stream.” It’s the same as specifying
“::$DATA” to the end of the file name. It has special behavior. As a
filter driver, though, I’m watching stream contexts (Vista does give you
file contexts and you can build them yourself - sort of - in prior
versions.) So here’s the weird part: opening the default data stream in
specific modes may delete the alternate data streams.

Here’s a specific example: if I open the “:Zone_Identifier:$DATA” stream
normally, and then I open the “::$DATA” stream with Overwrite
disposition, when I close the zone identifier stream, it will be
deleted. If I instead opened with supersede, the open would fail.

Thus, the stream could be deleted without anyone ever explicitly
deleting it…

But maybe you don’t care about streams being deleted. Maybe you only
care about files being deleted. I’m not sure what the behavior of
various dispositions would be when combined with ADS access, but I’ve
always assumed they will defy comprehension.

What does this all mean? It means that you cannot reliably know when
files (or streams) are deleted. A filter below you could change the
behavior. The order in which handles get closed may mean that one time
the file has been undeleted and another time it has not. If you query
on IRP_MJ_CLEANUP to see if it is delete pending, there is no guarantee
that there aren’t other opens on the file that won’t reverse that
decision (note: it doesn’t require another filter below you - even an
application can do this. Anyone building test suites that doesn’t test
this scenario is slacking.)

So every time I look at this I conclude: “cannot 100% reliably detect a
delete from a filter.” The suggestion always arises then that “well,
just look in the directory after we close the file and see if it really
went away.” Even a modicum of analysis will make you realize that this
just won’t work reliably (ergo, someone else races in and creates the
file while you’re checking in the directory.) One of the disadvantages
of living in a highly parallel environment.

Maybe “most of the time” is good enough for your filter (“it doesn’t
crash very often!”) I find getting the cases where I know it will work
correct, that I shy away from the cases where I know it cannot always
work.

Tony
OSR

Hi Lyndon,

Why go looking for yet more broken-ness? But yes, that’s an excellent
point - you could see a file opened, marked for delete, closed,
*deleted* and then have it magically reappear because the transaction
aborted.

So, the swamp just got worse. You are a bundle of good cheer Lyndon.
:wink:

Tony
OSR

> Hi Lyndon,

Why go looking for yet more broken-ness? But yes, that’s an
excellent point - you could see a file opened, marked for
delete, closed,
*deleted* and then have it magically reappear because the
transaction aborted.

So, the swamp just got worse. You are a bundle of good cheer Lyndon.
:wink:

If he weren’t so gleeful and chipper about it he’d be unbearable.

~Eric

This is all getting complicated because at all times you wish to be able to
depend on the DeletePending flag inside the FILE_OBJECT. Instead one should
do the bookkeeping himself and draw his conclusion according to the rules,
then it can be done properly.

If a create comes by with a FILE_DELETE_ON_CLOSE, you just ignore anything
else happening on the file object and wait till the close commes by then you
know your file is being deleted. If an IRP_MJ_SET_INFORMATION comes by then
monitor the DeleteFile flag. Watch that flag until close comes by and you
know whether that is going to delete the file or not.

If you need to take action on the file, because of a deletion, take action
immediately as soon as the IRP comes in because even at pre-cleanup stage
(or anything after) it’s too late to send operations down the stack. So in
the case you handle IRP_MJ_SET_INFORMATION you should keep information
around which allows you to undo the operation you did on the file object in
case that flag gets cleared in a future IRP.

Of course all this is happening on a first come first serve basis. At all
times your filter must be able to handle the fact that a file to be deleted
is no longer there.

//Daniel

“Tony Mason” wrote in message news:xxxxx@ntfsd…
> Detecting file deletion is a horrible mess from a mini-filter and,
> ultimately, cannot be 100% achieved correctly.
>
> Let’s look at FAT first (ergo, let’s pretend streams don’t exist.) If
> you open the file with FILE_DELETE_ON_CLOSE the attribute to delete the
> file is set in the CCB.
>
> {
> PCCB Ccb;
> Ccb = (PCCB)FileObject->FsContext2;
>
> //
> // Mark the DeleteOnClose bit if the operation was
> successful.
> //
>
> if ( DeleteOnClose ) {
>
> SetFlag( Ccb->Flags, CCB_FLAG_DELETE_ON_CLOSE );
> }
>
> If you open the file for DELETE access and then set (or clear) the
> disposition, the delete the file bit is set (or cleared) in the FCB:
>
> //
> // At this point either we have a file or an empty directory
> // so we know the delete can proceed.
> //
>
> SetFlag( Fcb->FcbState, FCB_STATE_DELETE_ON_CLOSE );
> FileObject->DeletePending = TRUE;
>
>
> The problem with this behavior is that it creates a situation in which
> the filter can never know if the file is really being deleted, since it
> can be “undeleted” right up to the last moment (e.g., it could be
> “undeleted” by a filter below you even.) However, this implementation
> is peculiar in that it means if you open it “DELETE_ON_CLOSE” an
> application can never directly reverse that decision (because set/clear
> disposition impacts the FCB and not the CCB.) The CCB bit is passed to
> the FCB on cleanup:
>
> //
> // Do a check here if this was a DELETE_ON_CLOSE FileObject,
> and
> // set the Fcb flag appropriately.
> //
>
> if (FlagOn(Ccb->Flags, CCB_FLAG_DELETE_ON_CLOSE)) {
>
> ASSERT( NodeType(Fcb) != FAT_NTC_ROOT_DCB );
>
> SetFlag(Fcb->FcbState, FCB_STATE_DELETE_ON_CLOSE);
>
> Of course, this leads to really crazy scenarios: an application opens
> the file (normally) and specifies shared delete semantics. It then
> opens the file (delete on close) in a consistent sharing manner.
> Depending upon whether or not the first or second handle gets closed,
> you may or may not see the file “stop being deleted”.
>
> I logged a bug against this about five years ago for several reasons:
> (1) it’s not the same as what happens in NTFS; (2) it’s almost
> impossible to explain to people why they see erratic behavior based upon
> “order in which the handle gets closed”; (3) it makes detecting deletion
> (and handling it) in a filter driver even more challenging (well,
> probably not, but at first blush it looks that way.)
>
> No doubt to spite me, the NTFS semantics (which were rational) have been
> changed in recent releases to match the FAT semantics (so we now have
> two irrational, “depends on the order of close” and “screw you filter
> drivers” approaches to the world.)
>
> Prior to the “let’s make NTFS behave more erratically and
> inconsistently” change, NTFS used the FCB for the “delete on close”
> option so an application could “change its mind” before the handle was
> closed in either case. Thus, it had a consistent and understandable
> behavior pattern…
>
>> Tony
> OSR
>
>

xxxxx@resplendence.com wrote:

If a create comes by with a FILE_DELETE_ON_CLOSE, you just ignore
anything else happening on the file object and wait till the close
commes by then you know your file is being deleted.
But the close could come literally two days later.

Matt

I could agree that “detecting file deletion ultimately, cannot be 100%
achieved correctly”. But unless the ultimate goal of your filter is to tell
for sure if files are really being removed from disk, this does not need to
be any kind of problem for a normal undelete filter driver. Note that with
or without your filter the situation is not different. An application never
can tell for sure if a file is truly being deleted or not.

If you take any situation “as is”, are ready to handle the presence or
absence of any file at any time it is surely possible to create an undelete
driver which never crashes regardless of filters above you or below your or
transactions. Another filter driver in the stack could defeat your
operations but that certainly can be dealt with gracefully.

One problem that is faced by developing an undelete driver is that by the
time you are “sure” that a file is going to be deleted as a result from
operations on your file object, it’s too late to take action. That can be
overcome by taking action immediately. The scenario in which files are being
deleted and undeleted by subsequent set information/disposition IRPs (such
as expolorer does, it “deletes” and “undeletes” files just to check if that
operation would succeed) can be dealt with by building in undo functionality
for your operations.

//Daniel

“Matt” wrote in message news:xxxxx@ntfsd…
> xxxxx@resplendence.com wrote:
>> If a create comes by with a FILE_DELETE_ON_CLOSE, you just ignore
>> anything else happening on the file object and wait till the close commes
>> by then you know your file is being deleted.
> But the close could come literally two days later.
>
> Matt
>

>>such as expolorer does, it “deletes” and “undeletes” files just to
check if that operation would succeed

Ah, the wonderful world of Explorer :stuck_out_tongue:

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@resplendence.com
Sent: 07 November 2008 11:56
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Detecting file deletion in minifilter

*** WARNING ***

This mail has originated outside your organization, either from an
external partner or the Global Internet.
Keep this in mind if you answer this message.

I could agree that “detecting file deletion ultimately, cannot be 100%
achieved correctly”. But unless the ultimate goal of your filter is to
tell for sure if files are really being removed from disk, this does not
need to be any kind of problem for a normal undelete filter driver. Note
that with or without your filter the situation is not different. An
application never can tell for sure if a file is truly being deleted or
not.

If you take any situation “as is”, are ready to handle the presence or
absence of any file at any time it is surely possible to create an
undelete driver which never crashes regardless of filters above you or
below your or transactions. Another filter driver in the stack could
defeat your operations but that certainly can be dealt with gracefully.

One problem that is faced by developing an undelete driver is that by
the time you are “sure” that a file is going to be deleted as a result
from operations on your file object, it’s too late to take action. That
can be overcome by taking action immediately. The scenario in which
files are being deleted and undeleted by subsequent set
information/disposition IRPs (such as expolorer does, it “deletes” and
“undeletes” files just to check if that operation would succeed) can be
dealt with by building in undo functionality for your operations.

//Daniel

“Matt” wrote in message news:xxxxx@ntfsd…
> xxxxx@resplendence.com wrote:
>> If a create comes by with a FILE_DELETE_ON_CLOSE, you just ignore
>> anything else happening on the file object and wait till the close
>> commes by then you know your file is being deleted.
> But the close could come literally two days later.
>
> Matt
>


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@baesystems.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.

Thanks to all.

To keep things simpler, initially I will just intercept create and cleanup path,

If DeletePending in FileObject is found true in pre and post cleanup ( with cleanup result as success), I will treat this as file got deleted.

I will check in Create Options for FILE_DELETE_ON_CLOSE in pre and post create path ( with create result as success) and treat it as file deletion.

I am logging these deleted filenames in database using service application. Before logging I will once again validate file is not present (in service) and on successful validation I will log filename to database.

In Later stage of development, I will check if an IRP_MJ_SET_INFORMATION comes then monitor the DeleteFile flag.

Does it looks OK ?

xxxxx@resplendence.com wrote:

Actually, I can’t argue with you… :frowning:

Regarding the term ‘detect’ - I was thinking more in the terms of
‘immediate’ notification. With code, I think of ‘detect’ as meaning
in ‘real time’.

Secondly, as I’ve thought about it, honestly, I don’t remember ever
timing the close on a file object that had a delete flag present. I’ve
actually never timed or tested this, so I don’t know if it would behave
like other close operations. Therefore, I need to retract that
last post for the time being.

I’m left wondering if IRP_Close will come immediately if a delete flag
is present? If so, perhaps that could be of use… :slight_smile:

Matt

I could agree that “detecting file deletion ultimately, cannot be 100%
achieved correctly”. But unless the ultimate goal of your filter is
to tell for sure if files are really being removed from disk, this
does not need to be any kind of problem for a normal undelete filter
driver. Note that with or without your filter the situation is not
different. An application never can tell for sure if a file is truly
being deleted or not.

If you take any situation “as is”, are ready to handle the presence or
absence of any file at any time it is surely possible to create an
undelete driver which never crashes regardless of filters above you or
below your or transactions. Another filter driver in the stack could
defeat your operations but that certainly can be dealt with gracefully.

One problem that is faced by developing an undelete driver is that by
the time you are “sure” that a file is going to be deleted as a result
from operations on your file object, it’s too late to take action.
That can be overcome by taking action immediately. The scenario in
which files are being deleted and undeleted by subsequent set
information/disposition IRPs (such as expolorer does, it “deletes” and
“undeletes” files just to check if that operation would succeed) can
be dealt with by building in undo functionality for your operations.

//Daniel

As was pointed out by Tony, a sure-fire way to tell if files are really
going to be deleted does not exist but I was thinking you were developing
some sort of undelete filter. Note that there are other ways to delete
files such as destructive renames (IRP_MJ_SET_INFORMATION with
FileRenameInformation and ReplaceIfExists) and many ways to get rid of files
contents, such as creates which overwrite or supersede.

//Daniel

wrote in message news:xxxxx@ntfsd…
> Thanks to all.
>
> To keep things simpler, initially I will just intercept create and cleanup
> path,
>
> If DeletePending in FileObject is found true in pre and post cleanup (
> with cleanup result as success), I will treat this as file got deleted.
>
> I will check in Create Options for FILE_DELETE_ON_CLOSE in pre and post
> create path ( with create result as success) and treat it as file
> deletion.
>
> I am logging these deleted filenames in database using service
> application. Before logging I will once again validate file is not present
> (in service) and on successful validation I will log filename to database.
>
> In Later stage of development, I will check if an IRP_MJ_SET_INFORMATION
> comes then monitor the DeleteFile flag.
>
> Does it looks OK ?
>

Overwrite, as I recall, doesn’t just get rid of the contents, it’ll
destroy a reparse point (and presumably EA’s as well?) too. Supersede I
don’t recall if that’s the case or not.

~Eric

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@resplendence.com
Sent: Friday, November 07, 2008 8:03 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Detecting file deletion in minifilter

As was pointed out by Tony, a sure-fire way to tell if files
are really going to be deleted does not exist but I was
thinking you were developing some sort of undelete filter.
Note that there are other ways to delete files such as
destructive renames (IRP_MJ_SET_INFORMATION with
FileRenameInformation and ReplaceIfExists) and many ways to
get rid of files contents, such as creates which overwrite
or supersede.

//Daniel

wrote in message news:xxxxx@ntfsd…
> > Thanks to all.
> >
> > To keep things simpler, initially I will just intercept create and
> > cleanup path,
> >
> > If DeletePending in FileObject is found true in pre and
> post cleanup (
> > with cleanup result as success), I will treat this as file
> got deleted.
> >
> > I will check in Create Options for FILE_DELETE_ON_CLOSE in pre and
> > post create path ( with create result as success) and treat
> it as file
> > deletion.
> >
> > I am logging these deleted filenames in database using service
> > application. Before logging I will once again validate file is not
> > present (in service) and on successful validation I will
> log filename to database.
> >
> > In Later stage of development, I will check if an
> > IRP_MJ_SET_INFORMATION comes then monitor the DeleteFile flag.
> >
> > Does it looks OK ?
> >
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> You are currently subscribed to ntfsd as:
> xxxxx@edsiohio.com To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>

Curious, you’ve replied to my post - which I am not seeing here …

“Tony Mason” wrote in message news:xxxxx@ntfsd…
Hi Lyndon,

Why go looking for yet more broken-ness? But yes, that’s an excellent
point - you could see a file opened, marked for delete, closed,
deleted and then have it magically reappear because the transaction
aborted.

So, the swamp just got worse. You are a bundle of good cheer Lyndon.
:wink:

Tony
OSR

Well, I certainly saw it and I believe others did as well.

Hmm, I recall a rumor as to one of the common causes of blindness. :wink:

It’s strange how e-mail does (or does not) flow at times.

Tony
OSR