file_overwrite deletes ads

Hi all,

I have a mini-filter which uses ADS to store data within certain files it
manages. However I’ve just run across a problem in testing whereby files
opened with a disposition of FILE_OVERWRITE will recreate the file, but
excluding the ADS which was originally part of the file.

I had a quick check to see if we receive any delete disposition info which I
could intercept, but this appears to be done internally in the ntfs driver
as part of the create routine. I checked the FastFat driver to see if it
confirmed my suspicions, but it wasn’t obvious as it just seems to flush the
cache and recreate the EA, and it obviously doesn’t have any ADS support to
delete.

Is there any way to stop my ADS from being deleted if this flag is used?
I was thinking of checking for the flag in my pre-create (and storing the
result in the stream handle context), removing it if found and letting the
call go through to the FSD as normal. Then in my post-create I would do the
overwrite steps myself, which would ensure the ADS is maintained. Removing
flags just feels like a nasty hack though… ?

Thanks,
Ged.

These are the semantics of FILE_OVERWRITE in NTFS: alternate streams are deleted.

Further, my recollection of what we found is that if the alternate stream is currently open (so foo:my_stream is open but foo::$DATA is not) that the FILE_OVERWRITE will succeed (so ::$DATA is truncated) and the alternate stream goes away when you close it. If you do the same thing with FILE_SUPERSEDE it also makes the ADS go away but fails if the alternate stream is open.

Could you simply grab the contents of your ADS in the overwrite path and then restore it in completion if the destructive create is successful? That’s likely less work.

Tony
OSR

On 03/06/2014 20:02, “Tony Mason” wrote:

>Further, my recollection of what we found is that if the alternate stream
>is currently open (so foo:my_stream is open but foo::$DATA is not) that
>the FILE_OVERWRITE will succeed (so ::$DATA is truncated) and the
>alternate stream goes away when you close it. If you do the same thing
>with FILE_SUPERSEDE it also makes the ADS go away but fails if the
>alternate stream is open.

I was surprised to see Microsoft=B9s Zone.Identifier streams disappear too,
considering it’s used as a sort of security notification feature

>Could you simply grab the contents of your ADS in the overwrite path and
>then restore it in completion if the destructive create is successful?
>That’s likely less work.

I thought about this approach but the ADS is a copy or reparse of the
primary stream, which means it could potentially be many gigabytes in
size. It=B9d mean creating a file and transferring the data from and back to
the original ADS, which will be very expensive.

I=B9m struggling to come up with any workable ideas other than removing the
flag, which I=B9m not happy about doing.

Ged.

I spent some time struggling with this too, eventually I changed my design
a little to deal with my ADS being removed. I couldn’t find a way to keep
the ADS short of removing the FILE_OVERWRITE and truncating the file in
postCreate.

However, it really depends how you think about the ADS. For example, if the
ADS contains some information about the data in the file (like for the Zone
Identifier) then you can safely remove the ADS on FILE_OVERWRITE because
all the file contents are gone. Also, from reading your comment, if your
ADS is a copy of the primary stream, perhaps it should go away when all the
data in the primary stream goes away? Or at least you could check in
preCreate whether there was an ADS and then in postCreate, after
FILE_OVERWRITE you would create a new, empty ADS (which is technically a
copy of new data in the file)…

Thanks,
Alex.

On Tue, Jun 3, 2014 at 12:32 PM, Ged Murphy
wrote:

> On 03/06/2014 20:02, “Tony Mason” wrote:
>
> >Further, my recollection of what we found is that if the alternate stream
> >is currently open (so foo:my_stream is open but foo::$DATA is not) that
> >the FILE_OVERWRITE will succeed (so ::$DATA is truncated) and the
> >alternate stream goes away when you close it. If you do the same thing
> >with FILE_SUPERSEDE it also makes the ADS go away but fails if the
> >alternate stream is open.
>
> I was surprised to see Microsoft¹s Zone.Identifier streams disappear too,
> considering it’s used as a sort of security notification feature
>
>
>
> >Could you simply grab the contents of your ADS in the overwrite path and
> >then restore it in completion if the destructive create is successful?
> >That’s likely less work.
>
>
> I thought about this approach but the ADS is a copy or reparse of the
> primary stream, which means it could potentially be many gigabytes in
> size. It¹d mean creating a file and transferring the data from and back to
> the original ADS, which will be very expensive.
>
> I¹m struggling to come up with any workable ideas other than removing the
> flag, which I¹m not happy about doing.
>
> Ged.
>
>
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

I just thought I’d update this thread with some final information in case anyone is searching for info on this in the future.

If I had a fancy blog like Alex (and the knowledge to go with it), I probably would have put it there, but this will have to do :slight_smile:

It seems that contrary to what the docs say, FILE_SUPERSEDE doesn’t actually replace the file, it just truncates it to zero. This means we can effectively replicate the functionality of all three dispositions (SUPERSEDE/OVERWRITE(_IF)) by resetting the file position and end of file information. In doing this, the only difference (I can see) between using the available disposition flags, or removing them and doing it yourself in the post-op is that the flags will do the whole operation within the NTFS driver, whereas doing it yourself will send two IRP_MJ_SET_INFORMATION irps down the stack

One area I was concerned about was the change notifications. The FastFat driver in the WDK samples shows that FILE_SUPERSEDE/OVERWRITE will invoke the following filters FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE

From my testing, it seems that the NTFS on x86 Win7 seems to do the same, however the NTFS driver on x64 Win8 only seems to invoke the FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_SIZE flags (The FILE_NOTIFY_CHANGE_LAST_WRITE flag is obviously only sent if the file wasn’t already empty).

In sending the FilePositionInformation and FileEndOfFileInformation to the FSD, we get exactly the same change notifications as we would do when using the provided dispositions.

Finally, a bit of brief sample code to show what’s needed.

(note: this is test code and hasn’t been used in a production environment. It may contain errors)

PreCreate

CreateDisposition = (UCHAR)(Data->Iopb->Parameters.Create.Options >> 24);

if (FileContext->ContainsStreams)
{
/* Check if the caller requested an overwrite */
if (CreateDisposition == FILE_SUPERSEDE ||
CreateDisposition == FILE_OVERWRITE ||
CreateDisposition == FILE_OVERWRITE_IF)
{
/*
* When a overwrite is done on a primary stream, all the ADS’s that are linked
* to that file are deleted in the NTFS driver.
* To fix this, we replace the overwrite flags with corresponding open flags before
* sending it to the FSD, and we do the overwrite work ourselves in the PostCreate
*/
RemovedOverwriteFlags = TRUE;

/* Replace the disposition flag with the corresponding open flags. SUPERSEDE and OVERWRITE_IF are synonymous */
TempCreateDisposition = (CreateDisposition == FILE_OVERWRITE) ? FILE_OPEN : FILE_OPEN_IF;

/* Clear out the old disposition bits */
Data->Iopb->Parameters.Create.Options &= 0x00FFFFFF;

/* Set the disposition which we’ll be sending to the FSD to stop the overwrite */
Data->Iopb->Parameters.Create.Options |= TempCreateDisposition << 24;

/* Mark that we have changed something */
FltSetCallbackDataDirty(Data);
}

/* Request a post-op */
*CallbackStatus = FLT_PREOP_SUCCESS_WITH_CALLBACK;
}

PostCreate

/* Check if we removed any of the overwrite flags in the pre-op */
if (RemovedOverwriteFlags)
{
FILE_POSITION_INFORMATION FilePosition;
FILE_END_OF_FILE_INFORMATION EndOfFile;

/*
* The FastFat driver in the WDK shows that supersede/overwrite will invoke
* the following notify filters:
* FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE;
* The NTFS driver on 32bit Win7 seems to do the same, however the NTFS drvier on x64 Win8 only
* seems to invoke the FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_SIZE flags
* (the FILE_NOTIFY_CHANGE_SIZE is only sent it the file wasn’t empty)
*/

/* Set the file position to zero */
FilePosition.CurrentByteOffset.QuadPart = 0;
Status = FltSetInformationFile(FltObjects->Instance,
FltObjects->FileObject,
&FilePosition,
sizeof(FILE_POSITION_INFORMATION),
FilePositionInformation);
if (!NT_SUCCESS(Status))
{
TRACE_ERROR(TraceHandle, “Failed to set the file position to zero : 0x%X”, Status);
return Status;
}

/* Set the end of file to zero */
EndOfFile.EndOfFile.QuadPart = 0;
Status = FltSetInformationFile(FltObjects->Instance,
FltObjects->FileObject,
&EndOfFile,
sizeof(FILE_END_OF_FILE_INFORMATION),
FileEndOfFileInformation);
if (!NT_SUCCESS(Status))
{
TRACE_ERROR(TraceHandle, “Failed to update the EOF : 0x%X”, Status);
return Status;
}
}

A few observations:

  1. Supersede and Overwrite are not completely interchangeable. Supersede
    replaces the attributes of the file where Overwrite preserves them. See the
    FASTFAT source:

//
// Modify the attributes and time of the file, by first reading
// in the dirent for the file and then updating its attributes
// and time fields. Note that for supersede we replace the file
// attributes as opposed to adding to them.
//

if (CreateDisposition == FILE_SUPERSEDE) {

Dirent->Attributes = FileAttributes;

} else {

Dirent->Attributes |= FileAttributes;

}

They also require different access permissions (Supersede requires DELETE,
Overwrite requires WRITE)

  1. You have potentially introduced an escalation of privilege with your
    changes. It appears that if a user with Read Only access to a file opened
    for Supersede your filter would demote the Supersede to an Open and then
    truncate the file on the user’s behalf.

  2. If you’re not running the IFS HCKs I highly suggest stopping now and
    getting them set up. They’re annoying strict about dispositions, access
    masks, and resulting behaviors (e.g. attributes preserved vs not preserved),
    which is exactly what the kind of validation you want when you’re changing
    things like this.

I suspect that the change in behavior that you’re introducing might also
cause the tests to fail, but at that point you have a business decision to
either knowingly not be eligible for logo or seek an exemption.

-scott
OSR
@OSRDrivers

“Ged Murphy” wrote in message
news:xxxxx@ntfsd…

I just thought I’d update this thread with some final information in case
anyone is searching for info on this in the future.

If I had a fancy blog like Alex (and the knowledge to go with it), I
probably would have put it there, but this will have to do J

It seems that contrary to what the docs say, FILE_SUPERSEDE doesn’t actually
replace the file, it just truncates it to zero. This means we can
effectively replicate the functionality of all three dispositions
(SUPERSEDE/OVERWRITE(_IF)) by resetting the file position and end of file
information. In doing this, the only difference (I can see) between using
the available disposition flags, or removing them and doing it yourself in
the post-op is that the flags will do the whole operation within the NTFS
driver, whereas doing it yourself will send two IRP_MJ_SET_INFORMATION irps
down the stack

One area I was concerned about was the change notifications. The FastFat
driver in the WDK samples shows that FILE_SUPERSEDE/OVERWRITE will invoke
the following filters FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE

From my testing, it seems that the NTFS on x86 Win7 seems to do the same,
however the NTFS driver on x64 Win8 only seems to invoke the
FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_SIZE flags (The
FILE_NOTIFY_CHANGE_LAST_WRITE flag is obviously only sent if the file wasn’t
already empty).

In sending the FilePositionInformation and FileEndOfFileInformation to the
FSD, we get exactly the same change notifications as we would do when using
the provided dispositions.

Finally, a bit of brief sample code to show what’s needed.

(note: this is test code and hasn’t been used in a production environment.
It may contain errors)

PreCreate

CreateDisposition = (UCHAR)(Data->Iopb->Parameters.Create.Options >> 24); if
(FileContext->ContainsStreams){ /* Check if the caller requested an
overwrite / if (CreateDisposition == FILE_SUPERSEDE ||
CreateDisposition == FILE_OVERWRITE || CreateDisposition ==
FILE_OVERWRITE_IF) { /
* When a overwrite is done on a
primary stream, all the ADS’s that are linked * to that file are
deleted in the NTFS driver. * To fix this, we replace the overwrite
flags with corresponding open flags before * sending it to the FSD,
and we do the overwrite work ourselves in the PostCreate /
RemovedOverwriteFlags = TRUE; /
Replace the disposition flag with
the corresponding open flags. SUPERSEDE and OVERWRITE_IF are synonymous /
TempCreateDisposition = (CreateDisposition == FILE_OVERWRITE) ? FILE_OPEN :
FILE_OPEN_IF; /
Clear out the old disposition bits /
Data->Iopb->Parameters.Create.Options &= 0x00FFFFFF; /
Set the
disposition which we’ll be sending to the FSD to stop the overwrite /
Data->Iopb->Parameters.Create.Options |= TempCreateDisposition << 24;
/
Mark that we have changed something /
FltSetCallbackDataDirty(Data); } /
Request a post-op /
CallbackStatus = FLT_PREOP_SUCCESS_WITH_CALLBACK;} PostCreate / Check if
we removed any of the overwrite flags in the pre-op /if
(RemovedOverwriteFlags){ FILE_POSITION_INFORMATION FilePosition;
FILE_END_OF_FILE_INFORMATION EndOfFile; /
* The FastFat driver in
the WDK shows that supersede/overwrite will invoke * the following
notify filters: * FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE; * The NTFS
driver on 32bit Win7 seems to do the same, however the NTFS drvier on x64
Win8 only * seems to invoke the FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_SIZE flags * (the FILE_NOTIFY_CHANGE_SIZE is only
sent it the file wasn’t empty) / / Set the file position to zero
/ FilePosition.CurrentByteOffset.QuadPart = 0; Status =
FltSetInformationFile(FltObjects->Instance,
FltObjects->FileObject, &FilePosition,
sizeof(FILE_POSITION_INFORMATION),
FilePositionInformation); if (!NT_SUCCESS(Status)) {
TRACE_ERROR(TraceHandle, “Failed to set the file position to zero : 0x%X”,
Status); return Status; } /* Set the end of file to zero */
EndOfFile.EndOfFile.QuadPart = 0; Status =
FltSetInformationFile(FltObjects->Instance,
FltObjects->FileObject, &EndOfFile,
sizeof(FILE_END_OF_FILE_INFORMATION),
FileEndOfFileInformation); if (!NT_SUCCESS(Status)) {
TRACE_ERROR(TraceHandle, “Failed to update the EOF : 0x%X”, Status);
return Status; }}

Hi Scott,
Thanks for your reply, that’s really useful and I’d certainly overlooked a few things.

  1. Interesting, I must have missed that in my preliminary tests. Perhaps I got something mixed up in the tests and that would explain the different results I mentioned I was seeing in Win7 and Win8. I’ll take a look at that area again.

  2. That’s a good point. I do have some access checks in my actual code (not in the code I posted), but they don’t take into account the read only issue you pointed out. Looking at the FastFat code, most of the access checks which MSDN specifies are required actually aren’t, for example :
    “The CreateDisposition value FILE_SUPERSEDE requires that the caller have DELETE access to an existing file object”

The FSD just removes them and adds them anyway

} else if (CreateDisposition == FILE_SUPERSEDE) {

SetFlag( AddedAccess,
DELETE & ~(*DesiredAccess) );

*DesiredAccess |= DELETE;

} else if ((CreateDisposition == FILE_OVERWRITE) ||
(CreateDisposition == FILE_OVERWRITE_IF)) {

SetFlag( AddedAccess,
(FILE_WRITE_DATA | FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES) & ~(*DesiredAccess) );

*DesiredAccess |= FILE_WRITE_DATA | FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES;
}

  1. I’m not currently running the HCK, admittedly I wasn’t aware of it. It sounds really useful for this scenario though, thanks for the tip.

Additionally, I also have a decision to make about what to do with any other streams which may be part of the file. If I want to copy how the NTFS driver works more closely, I should consider deleting all the other streams which aren’t ‘managed’. This is what the user may be expecting when they supply one of those flags.

It seems I was a little premature in posting the code, especially as I’m still in the process of writing tests. However Alex has kindly suggested I do a write up for his blog once I’ve got everything nailed, so I’ll hopefully be able to put something useful together once I’m happy with everything.

Thanks,
Ged.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: 06 June 2014 15:57
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] file_overwrite deletes ads

A few observations:

  1. Supersede and Overwrite are not completely interchangeable. Supersede replaces the attributes of the file where Overwrite preserves them. See the FASTFAT source:

//
// Modify the attributes and time of the file, by first reading
// in the dirent for the file and then updating its attributes
// and time fields. Note that for supersede we replace the file
// attributes as opposed to adding to them.
//

if (CreateDisposition == FILE_SUPERSEDE) {

Dirent->Attributes = FileAttributes;

} else {

Dirent->Attributes |= FileAttributes;

}

They also require different access permissions (Supersede requires DELETE, Overwrite requires WRITE)

  1. You have potentially introduced an escalation of privilege with your changes. It appears that if a user with Read Only access to a file opened for Supersede your filter would demote the Supersede to an Open and then truncate the file on the user’s behalf.

  2. If you’re not running the IFS HCKs I highly suggest stopping now and getting them set up. They’re annoying strict about dispositions, access masks, and resulting behaviors (e.g. attributes preserved vs not preserved), which is exactly what the kind of validation you want when you’re changing things like this.

I suspect that the change in behavior that you’re introducing might also cause the tests to fail, but at that point you have a business decision to either knowingly not be eligible for logo or seek an exemption.

-scott
OSR
@OSRDrivers

“Ged Murphy” wrote in message news:xxxxx@ntfsd…

I just thought I’d update this thread with some final information in case anyone is searching for info on this in the future.

If I had a fancy blog like Alex (and the knowledge to go with it), I probably would have put it there, but this will have to do J

It seems that contrary to what the docs say, FILE_SUPERSEDE doesn’t actually
replace the file, it just truncates it to zero. This means we can
effectively replicate the functionality of all three dispositions
(SUPERSEDE/OVERWRITE(_IF)) by resetting the file position and end of file
information. In doing this, the only difference (I can see) between using
the available disposition flags, or removing them and doing it yourself in
the post-op is that the flags will do the whole operation within the NTFS
driver, whereas doing it yourself will send two IRP_MJ_SET_INFORMATION irps
down the stack

One area I was concerned about was the change notifications. The FastFat
driver in the WDK samples shows that FILE_SUPERSEDE/OVERWRITE will invoke
the following filters FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE

From my testing, it seems that the NTFS on x86 Win7 seems to do the same,
however the NTFS driver on x64 Win8 only seems to invoke the
FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_SIZE flags (The
FILE_NOTIFY_CHANGE_LAST_WRITE flag is obviously only sent if the file wasn’t
already empty).

In sending the FilePositionInformation and FileEndOfFileInformation to the
FSD, we get exactly the same change notifications as we would do when using
the provided dispositions.

Finally, a bit of brief sample code to show what’s needed.

(note: this is test code and hasn’t been used in a production environment.
It may contain errors)

PreCreate

CreateDisposition = (UCHAR)(Data->Iopb->Parameters.Create.Options >> 24); if
(FileContext->ContainsStreams){ /* Check if the caller requested an
overwrite / if (CreateDisposition == FILE_SUPERSEDE ||
CreateDisposition == FILE_OVERWRITE || CreateDisposition ==
FILE_OVERWRITE_IF) { /
* When a overwrite is done on a
primary stream, all the ADS’s that are linked * to that file are
deleted in the NTFS driver. * To fix this, we replace the overwrite
flags with corresponding open flags before * sending it to the FSD,
and we do the overwrite work ourselves in the PostCreate /
RemovedOverwriteFlags = TRUE; /
Replace the disposition flag with
the corresponding open flags. SUPERSEDE and OVERWRITE_IF are synonymous /
TempCreateDisposition = (CreateDisposition == FILE_OVERWRITE) ? FILE_OPEN :
FILE_OPEN_IF; /
Clear out the old disposition bits /
Data->Iopb->Parameters.Create.Options &= 0x00FFFFFF; /
Set the
disposition which we’ll be sending to the FSD to stop the overwrite /
Data->Iopb->Parameters.Create.Options |= TempCreateDisposition << 24;
/
Mark that we have changed something /
FltSetCallbackDataDirty(Data); } /
Request a post-op /
CallbackStatus = FLT_PREOP_SUCCESS_WITH_CALLBACK;} PostCreate / Check if
we removed any of the overwrite flags in the pre-op /if
(RemovedOverwriteFlags){ FILE_POSITION_INFORMATION FilePosition;
FILE_END_OF_FILE_INFORMATION EndOfFile; /
* The FastFat driver in
the WDK shows that supersede/overwrite will invoke * the following
notify filters: * FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE; * The NTFS
driver on 32bit Win7 seems to do the same, however the NTFS drvier on x64
Win8 only * seems to invoke the FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_SIZE flags * (the FILE_NOTIFY_CHANGE_SIZE is only
sent it the file wasn’t empty) / / Set the file position to zero
/ FilePosition.CurrentByteOffset.QuadPart = 0; Status =
FltSetInformationFile(FltObjects->Instance,
FltObjects->FileObject, &FilePosition,
sizeof(FILE_POSITION_INFORMATION),
FilePositionInformation); if (!NT_SUCCESS(Status)) {
TRACE_ERROR(TraceHandle, “Failed to set the file position to zero : 0x%X”,
Status); return Status; } /* Set the end of file to zero */
EndOfFile.EndOfFile.QuadPart = 0; Status =
FltSetInformationFile(FltObjects->Instance,
FltObjects->FileObject, &EndOfFile,
sizeof(FILE_END_OF_FILE_INFORMATION),
FileEndOfFileInformation); if (!NT_SUCCESS(Status)) {
TRACE_ERROR(TraceHandle, “Failed to update the EOF : 0x%X”, Status);
return Status; }}


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Unfortunately FAT doesn’t actually support security, so how FAT handles
these things will differ than how NTFS/ReFS do.

Note that FAT there is also additional enforcement of ACCESS_SYSTEM_SECURITY
around Supersede/Overwrite (see FatCheckSystemSecurityAccess). Something
else you want to make sure you don’t inadvertently allow the caller to
bypass.

(Are we having fun yet :))

-scott
OSR
@OSRDrivers

“Ged Murphy” wrote in message news:xxxxx@ntfsd…

Hi Scott,
Thanks for your reply, that’s really useful and I’d certainly overlooked a
few things.

  1. Interesting, I must have missed that in my preliminary tests. Perhaps I
    got something mixed up in the tests and that would explain the different
    results I mentioned I was seeing in Win7 and Win8. I’ll take a look at that
    area again.

  2. That’s a good point. I do have some access checks in my actual code (not
    in the code I posted), but they don’t take into account the read only issue
    you pointed out. Looking at the FastFat code, most of the access checks
    which MSDN specifies are required actually aren’t, for example :
    “The CreateDisposition value FILE_SUPERSEDE requires that the caller have
    DELETE access to an existing file object”

The FSD just removes them and adds them anyway

} else if (CreateDisposition == FILE_SUPERSEDE) {

SetFlag( AddedAccess,
DELETE & ~(*DesiredAccess) );

*DesiredAccess |= DELETE;

} else if ((CreateDisposition == FILE_OVERWRITE) ||
(CreateDisposition == FILE_OVERWRITE_IF)) {

SetFlag( AddedAccess,
(FILE_WRITE_DATA | FILE_WRITE_EA |
FILE_WRITE_ATTRIBUTES) & ~(*DesiredAccess) );

*DesiredAccess |= FILE_WRITE_DATA | FILE_WRITE_EA |
FILE_WRITE_ATTRIBUTES;
}

  1. I’m not currently running the HCK, admittedly I wasn’t aware of it. It
    sounds really useful for this scenario though, thanks for the tip.

Additionally, I also have a decision to make about what to do with any other
streams which may be part of the file. If I want to copy how the NTFS driver
works more closely, I should consider deleting all the other streams which
aren’t ‘managed’. This is what the user may be expecting when they supply
one of those flags.

It seems I was a little premature in posting the code, especially as I’m
still in the process of writing tests. However Alex has kindly suggested I
do a write up for his blog once I’ve got everything nailed, so I’ll
hopefully be able to put something useful together once I’m happy with
everything.

Thanks,
Ged.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: 06 June 2014 15:57
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] file_overwrite deletes ads

A few observations:

  1. Supersede and Overwrite are not completely interchangeable. Supersede
    replaces the attributes of the file where Overwrite preserves them. See the
    FASTFAT source:

//
// Modify the attributes and time of the file, by first reading
// in the dirent for the file and then updating its attributes
// and time fields. Note that for supersede we replace the file
// attributes as opposed to adding to them.
//

if (CreateDisposition == FILE_SUPERSEDE) {

Dirent->Attributes = FileAttributes;

} else {

Dirent->Attributes |= FileAttributes;

}

They also require different access permissions (Supersede requires DELETE,
Overwrite requires WRITE)

  1. You have potentially introduced an escalation of privilege with your
    changes. It appears that if a user with Read Only access to a file opened
    for Supersede your filter would demote the Supersede to an Open and then
    truncate the file on the user’s behalf.

  2. If you’re not running the IFS HCKs I highly suggest stopping now and
    getting them set up. They’re annoying strict about dispositions, access
    masks, and resulting behaviors (e.g. attributes preserved vs not preserved),
    which is exactly what the kind of validation you want when you’re changing
    things like this.

I suspect that the change in behavior that you’re introducing might also
cause the tests to fail, but at that point you have a business decision to
either knowingly not be eligible for logo or seek an exemption.

-scott
OSR
@OSRDrivers

“Ged Murphy” wrote in message
news:xxxxx@ntfsd…

I just thought I’d update this thread with some final information in case
anyone is searching for info on this in the future.

If I had a fancy blog like Alex (and the knowledge to go with it), I
probably would have put it there, but this will have to do J

It seems that contrary to what the docs say, FILE_SUPERSEDE doesn’t actually
replace the file, it just truncates it to zero. This means we can
effectively replicate the functionality of all three dispositions
(SUPERSEDE/OVERWRITE(_IF)) by resetting the file position and end of file
information. In doing this, the only difference (I can see) between using
the available disposition flags, or removing them and doing it yourself in
the post-op is that the flags will do the whole operation within the NTFS
driver, whereas doing it yourself will send two IRP_MJ_SET_INFORMATION irps
down the stack

One area I was concerned about was the change notifications. The FastFat
driver in the WDK samples shows that FILE_SUPERSEDE/OVERWRITE will invoke
the following filters FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE

From my testing, it seems that the NTFS on x86 Win7 seems to do the same,
however the NTFS driver on x64 Win8 only seems to invoke the
FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_SIZE flags (The
FILE_NOTIFY_CHANGE_LAST_WRITE flag is obviously only sent if the file wasn’t
already empty).

In sending the FilePositionInformation and FileEndOfFileInformation to the
FSD, we get exactly the same change notifications as we would do when using
the provided dispositions.

Finally, a bit of brief sample code to show what’s needed.

(note: this is test code and hasn’t been used in a production environment.
It may contain errors)

PreCreate

CreateDisposition = (UCHAR)(Data->Iopb->Parameters.Create.Options >> 24); if
(FileContext->ContainsStreams){ /* Check if the caller requested an
overwrite / if (CreateDisposition == FILE_SUPERSEDE ||
CreateDisposition == FILE_OVERWRITE || CreateDisposition ==
FILE_OVERWRITE_IF) { /
* When a overwrite is done on a
primary stream, all the ADS’s that are linked * to that file are
deleted in the NTFS driver. * To fix this, we replace the overwrite
flags with corresponding open flags before * sending it to the FSD,
and we do the overwrite work ourselves in the PostCreate /
RemovedOverwriteFlags = TRUE; /
Replace the disposition flag with
the corresponding open flags. SUPERSEDE and OVERWRITE_IF are synonymous /
TempCreateDisposition = (CreateDisposition == FILE_OVERWRITE) ? FILE_OPEN :
FILE_OPEN_IF; /
Clear out the old disposition bits /
Data->Iopb->Parameters.Create.Options &= 0x00FFFFFF; /
Set the
disposition which we’ll be sending to the FSD to stop the overwrite /
Data->Iopb->Parameters.Create.Options |= TempCreateDisposition << 24;
/
Mark that we have changed something /
FltSetCallbackDataDirty(Data); } /
Request a post-op /
CallbackStatus = FLT_PREOP_SUCCESS_WITH_CALLBACK;} PostCreate / Check if
we removed any of the overwrite flags in the pre-op /if
(RemovedOverwriteFlags){ FILE_POSITION_INFORMATION FilePosition;
FILE_END_OF_FILE_INFORMATION EndOfFile; /
* The FastFat driver in
the WDK shows that supersede/overwrite will invoke * the following
notify filters: * FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE; * The NTFS
driver on 32bit Win7 seems to do the same, however the NTFS drvier on x64
Win8 only * seems to invoke the FILE_NOTIFY_CHANGE_LAST_WRITE |
FILE_NOTIFY_CHANGE_SIZE flags * (the FILE_NOTIFY_CHANGE_SIZE is only
sent it the file wasn’t empty) / / Set the file position to zero
/ FilePosition.CurrentByteOffset.QuadPart = 0; Status =
FltSetInformationFile(FltObjects->Instance,
FltObjects->FileObject, &FilePosition,
sizeof(FILE_POSITION_INFORMATION),
FilePositionInformation); if (!NT_SUCCESS(Status)) {
TRACE_ERROR(TraceHandle, “Failed to set the file position to zero : 0x%X”,
Status); return Status; } /* Set the end of file to zero */
EndOfFile.EndOfFile.QuadPart = 0; Status =
FltSetInformationFile(FltObjects->Instance,
FltObjects->FileObject, &EndOfFile,
sizeof(FILE_END_OF_FILE_INFORMATION),
FileEndOfFileInformation); if (!NT_SUCCESS(Status)) {
TRACE_ERROR(TraceHandle, “Failed to update the EOF : 0x%X”, Status);
return Status; }}


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

The read only bit is not a security attribute. It’s a file attribute. Thus, as long as the caller can change the attributes (FILE_WRITE_ATTRIBUTES), they can clear read-only. But neither the OS nor the file system will NOT do it for them automatically. Changing the size of a file (EOF, VDL, Allocation) requires FILE_WRITE_DATA. Changing the attributes requires FILE_WRITE_ATTRIBUTES. This is enforced by the I/O Manager, not the file system, so it’s not NTFS specific.

Supersede requires DELETE access, which isn’t impacted by the read-only bit. Overwrite doesn’t require delete access and IS impacted by the read-only bit.

On the flip side, the read only bit is not a security attribute. So even if FILE_WRITE_ATTRIBUTES is granted as part of the file access, the file systems will typically reject an open for write on the file. Thus, read-only is enforced by the FSD, while the granted security (which is associated with the given handle) is enforced by the I/O Manager.

Tangent: the read only bit is also a great way to screw up filter drivers. I can create a new file, ask for FILE_WRITE_DATA access but set the FILE_ATTRIBUTE_READONLY bit. The write access is granted, but now any subsequent open of the file for write will fail - including an attempt to write your ADS.

Tony
OSR

Another useful note from http://www.osronline.com/showThread.cfm?link=27213

I think that makes it clearer as to what happens. In particular, it probably changes the FileId, but I am not really sure (and have not tested it). Does anyone know?

  • Danilo

>I was surprised to see Microsoft?s Zone.Identifier streams disappear too,

Is this the thing which causes ZIP files from email attachments to ask a security question on open?

To get rid of the security warning, you can use GNU “cat” tool:

copy t.zip copy.zip
cat < copy.zip > t.zip

This is an empirical fact.

Since “cat” opens the target with OVERWRITE I think and is unaware of ADS, you can see that this is enough to delete all ADSes on a file.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Yet another idea:

  • ADS are evil :slight_smile:
  • instead, just use files in your special subdirectories whose names start with “.”, like SVN or Git use, and yes, they keep at least 1 alternate copy of each file in these subdirs
  • this will give you ReFS compat and also a possibility of a Linux port


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

“Ged Murphy” wrote in message news:xxxxx@ntfsd…
Yeah that’s seems to be the problem, it’s not consistent between the primary stream and the ADS’s. My goal now is to make It consistent by making the opening the primary stream with SUPERSEDE/OVERWRITE match the behaviour of ADS’s.

I hadn’t thought about making unmanaged apps use a stream. It’d probably work well in principle, but it’d cause a complete mess if our software was uninstalled as there’d be lots of files with their data suddenly missing.

Ged.

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Alex Carp
Sent: 04 June 2014 16:47
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Re: [ntfsd] file_overwrite deletes ads

Well, based on your description, it seems to me that the problem comes from the fact that some open modes (and operations) apply to the file as a whole instead of just one stream (delete, FILE_SUPERSEDE, FILE_OVERWRITE etc…) so the streams are not symmetrical (the default one can impact the managed one but not the other way around).

Have you considered making “unmanaged” applications also use a stream? For example, for foo.txt, you’d have foo.txt:managed and foo.txt:unmanaged and just reparse any unmanaged open to foo.txt:unmanaged. This way the two streams are symmetrical and deleting and overwriting one won’t affect the other ? Of course, this would mean that your filter must always be running and the distinction between managed and unmanaged disappears (or at least becomes very blurry).

Thanks,

Alex.

On Wed, Jun 4, 2014 at 2:13 AM, wrote:

Hi Marion,

Yes, other then the file location, there’s no correlation between the contents of the primary/non-managed and managed streams.

Okay, so based on the advise, I’m going to go with removing the FILE_OVERWRITE(_IF) flags (on the primary stream) and implementing this functionality in my post-create.

I’ll do some testing on NTFS to work out exactly what it does for overwrites, but it’ll most likely be similar to the FastFat copy, in that I’ll need to purge the file (and possibly the section cache), update the file attributes and report a notification change.

Thanks to Tony, Alex and Marion

Ged.

From: Marion Bond
Sent: ‎Wednesday‎, ‎4‎ ‎June‎ ‎2014 ‎01‎:‎07

To: Windows File Systems Devs Interest List

Are you saying that there is no correlation between the contents of the managed view and the unmanaged view? If that is the case, then you should certainly intercept the overwrite and perform your own behaviour. This would not be a hack, but merely the correct way to maintain the independence between the data in multiple streams

If, on the other hand, the data in these streams has some correlation, and I find it hard to believe that there is any useful scenario where they do not, then you need to define the protocol by which they interact. I have the suspicion that you are trying to optimize the case where an application writes almost the same data, and the question is whether you can detect that case any more quickly than you can recomputed the result.

Sent from Surface Pro

From: Ged Murphy
Sent: ‎Tuesday‎, ‎June‎ ‎03‎, ‎2014 ‎7‎:‎19‎ ‎PM
To: Windows File Systems Devs Interest List

Sorry guys, it seems I didn’t make it very clear as to the goal of the design.

Essentially I have two ways of running all applications on the box,. An app can either run in a managed mode (whereby it’s managed by our software) or an unmanaged mode.

When running in unmanaged mode, apps interact with files in a normal way. When running in managed mode, we reparse file creates on primary streams to managed streams and all data is written and read to that managed stream. Additional attributes can be added to managed streams, such as encryption or compression. As far as managed apps are concerned, they still think they’re accessing the primary stream.

Therefore, notepad++ running in normal mode will see one view of a file, but notepad++ running in managed mode will see an entirely different view of the file.

The problem arises when a non-managed app opens a primary stream with FILE_OVERWRITE. This doesn’t get reparsed to the managed stream, so the overwrite is done on the main stream which removes all the ADS’ and essentially deletes the view of the file which managed apps see.

This was one of those ‘oh sh*t’ moments when you realise your design has a hole and any possible fix dirties what was a previously clean solution….

Ged.

On 04/06/2014 00:00, “Alex Carp” wrote:

Yeah, like I said before, I’ve mostly used ADS as a mechanism to store metadata about file contents, so when the contents went away the metadata was no longer accurate (and necessary). However, perhaps Ged is implementing something like file versions, where all the old contents of the file are preserved in ADS or some such.

However, I wonder if the file name comes into play at all. For example, for a file version system, one should probably take into account the file name for a specific version as well, so maybe ADS isn’t a good choice for things that need to be tracked across renames… Anyway, I’m really just guessing here, Ged didn’t say anything more specific…

Thanks,

Alex.

On Tue, Jun 3, 2014 at 3:20 PM, Marion Bond wrote:

Interesting. You made it sound like the file you are working with are some kind of specially organized databases or something like that rather than any arbitrary file. I am curious what kind of meta data you might want to persist across a complete replacement of the contents that wasn’t triggered in some controlled way you control. In your example, nothing would stop me from pasting in an entirely different document in Wordpad and then saving. Replication information might be useful after something like that, but it shouldn’t occupy GB. Backup and restore data might occupy GB, but using alternate data streams seems like a poor way to implement it - especially as VSS and even recycle bin functionality already exists in the base OS

Sent from Surface Pro

From: Ged Murphy
Sent: Tuesday, June 03, 2014 6:07 PM

To: Windows File Systems Devs Interest List

The IO can come from any application on the box, so it could come at any time.

Wordpad is an example of an app which uses this flag when opening files for writing

On 03/06/2014 22:55, “Marion Bond” wrote:

How often do you expect this operation in real life? If the file IO is from your application, I would expect it not to do this without a good reason. And if the file IO is from another application (ie a restore by a backup application), then maybe regenerating multiple GB of data isn’t such a bad thing

Sent from Surface Pro


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTFSD is sponsored by OSR OSR is hiring!! Info at http://www.osr.com/careers For our schedule of debugging and file system seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTFSD is sponsored by OSR OSR is hiring!! Info at http://www.osr.com/careers For our schedule of debugging and file system seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[quote]
    - ADS are evil :slight_smile:
    - instead, just use files in your special subdirectories whose names start with “.”, like SVN or Git use, and yes, they keep at least 1 alternate copy of each file in these subdirs
    - this will give you ReFS compat and also a possibility of a Linux port
[/quote]  

ADS are just a tool, neither good nor ill. ADS have the benefit of actually associating with the file, so that the ADS doesn’t get separated from the original file - they attempted to generalize the idea of Extended Attributes.

I’m not sure how this gives you ReFS support - ReFS in Server 2012R2 supports ADS (http://en.wikipedia.org/wiki/ReFS) so using them wouldn’t preclude anything but support on Server 2012, which hardly seems to be much of a stumbling block.

Tony
OSR

ADS’ are a really neat solution for what we’re trying to solve, which is to have two views of the same file dependant on configuration.

I did look into separating the data into two files. I was actually going to hide the linked file completely to avoid any confusion with users wondering what these new files were. However it caused more issues than it solved.

The ADS solution does have its issues, but for our solution, it solves more than it creates, which is why I went with that.

I had looked into what it would mean for ReFS, and back in the design stages we were discussing implementing ADS on top of ReFS if required. However ADS has since been sneaked into Win8.1 for ReFS, so that solved that future potential issue for us :slight_smile:

Ged.

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: 09 June 2014 05:30
To: Windows File Systems Devs Interest List
Subject: Re: Re: [ntfsd] file_overwrite deletes ads

Yet another idea:

  • ADS are evil :slight_smile:

  • instead, just use files in your special subdirectories whose names start with “.”, like SVN or Git use, and yes, they keep at least 1 alternate copy of each file in these subdirs

  • this will give you ReFS compat and also a possibility of a Linux port


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com mailto:xxxxx
http://www.storagecraft.com

“Ged Murphy” > wrote in message news:xxxxx@ntfsd…

Yeah that’s seems to be the problem, it’s not consistent between the primary stream and the ADS’s. My goal now is to make It consistent by making the opening the primary stream with SUPERSEDE/OVERWRITE match the behaviour of ADS’s.

I hadn’t thought about making unmanaged apps use a stream. It’d probably work well in principle, but it’d cause a complete mess if our software was uninstalled as there’d be lots of files with their data suddenly missing.

Ged.

From: xxxxx@lists.osr.com mailto:xxxxx [mailto:xxxxx@lists.osr.com] On Behalf Of Alex Carp
Sent: 04 June 2014 16:47
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Re: [ntfsd] file_overwrite deletes ads

Well, based on your description, it seems to me that the problem comes from the fact that some open modes (and operations) apply to the file as a whole instead of just one stream (delete, FILE_SUPERSEDE, FILE_OVERWRITE etc…) so the streams are not symmetrical (the default one can impact the managed one but not the other way around).

Have you considered making “unmanaged” applications also use a stream? For example, for foo.txt, you’d have foo.txt:managed and foo.txt:unmanaged and just reparse any unmanaged open to foo.txt:unmanaged. This way the two streams are symmetrical and deleting and overwriting one won’t affect the other ? Of course, this would mean that your filter must always be running and the distinction between managed and unmanaged disappears (or at least becomes very blurry).

Thanks,

Alex.

On Wed, Jun 4, 2014 at 2:13 AM, > wrote:

Hi Marion,

Yes, other then the file location, there’s no correlation between the contents of the primary/non-managed and managed streams.

Okay, so based on the advise, I’m going to go with removing the FILE_OVERWRITE(_IF) flags (on the primary stream) and implementing this functionality in my post-create.

I’ll do some testing on NTFS to work out exactly what it does for overwrites, but it’ll most likely be similar to the FastFat copy, in that I’ll need to purge the file (and possibly the section cache), update the file attributes and report a notification change.

Thanks to Tony, Alex and Marion

Ged.

From: Marion Bond mailto:xxxxx
Sent: ‎Wednesday‎, ‎4‎ ‎June‎ ‎2014 ‎01‎:‎07

To: Windows File Systems Devs Interest List mailto:xxxxx

Are you saying that there is no correlation between the contents of the managed view and the unmanaged view? If that is the case, then you should certainly intercept the overwrite and perform your own behaviour. This would not be a hack, but merely the correct way to maintain the independence between the data in multiple streams

If, on the other hand, the data in these streams has some correlation, and I find it hard to believe that there is any useful scenario where they do not, then you need to define the protocol by which they interact. I have the suspicion that you are trying to optimize the case where an application writes almost the same data, and the question is whether you can detect that case any more quickly than you can recomputed the result.

Sent from Surface Pro

From: Ged Murphy mailto:xxxxx
Sent: ‎Tuesday‎, ‎June‎ ‎03‎, ‎2014 ‎7‎:‎19‎ ‎PM
To: Windows File Systems Devs Interest List mailto:xxxxx

Sorry guys, it seems I didn’t make it very clear as to the goal of the design.

Essentially I have two ways of running all applications on the box,. An app can either run in a managed mode (whereby it’s managed by our software) or an unmanaged mode.

When running in unmanaged mode, apps interact with files in a normal way. When running in managed mode, we reparse file creates on primary streams to managed streams and all data is written and read to that managed stream. Additional attributes can be added to managed streams, such as encryption or compression. As far as managed apps are concerned, they still think they’re accessing the primary stream.

Therefore, notepad++ running in normal mode will see one view of a file, but notepad++ running in managed mode will see an entirely different view of the file.

The problem arises when a non-managed app opens a primary stream with FILE_OVERWRITE. This doesn’t get reparsed to the managed stream, so the overwrite is done on the main stream which removes all the ADS’ and essentially deletes the view of the file which managed apps see.

This was one of those ‘oh sh*t’ moments when you realise your design has a hole and any possible fix dirties what was a previously clean solution….

Ged.

On 04/06/2014 00:00, “Alex Carp” > wrote:

Yeah, like I said before, I’ve mostly used ADS as a mechanism to store metadata about file contents, so when the contents went away the metadata was no longer accurate (and necessary). However, perhaps Ged is implementing something like file versions, where all the old contents of the file are preserved in ADS or some such.

However, I wonder if the file name comes into play at all. For example, for a file version system, one should probably take into account the file name for a specific version as well, so maybe ADS isn’t a good choice for things that need to be tracked across renames… Anyway, I’m really just guessing here, Ged didn’t say anything more specific…

Thanks,

Alex.

On Tue, Jun 3, 2014 at 3:20 PM, Marion Bond > wrote:

Interesting. You made it sound like the file you are working with are some kind of specially organized databases or something like that rather than any arbitrary file. I am curious what kind of meta data you might want to persist across a complete replacement of the contents that wasn’t triggered in some controlled way you control. In your example, nothing would stop me from pasting in an entirely different document in Wordpad and then saving. Replication information might be useful after something like that, but it shouldn’t occupy GB. Backup and restore data might occupy GB, but using alternate data streams seems like a poor way to implement it - especially as VSS and even recycle bin functionality already exists in the base OS

Sent from Surface Pro

From: Ged Murphy mailto:xxxxx
Sent: Tuesday, June 03, 2014 6:07 PM

To: Windows File Systems Devs Interest List mailto:xxxxx

The IO can come from any application on the box, so it could come at any time.

Wordpad is an example of an app which uses this flag when opening files for writing

On 03/06/2014 22:55, “Marion Bond” > wrote:

How often do you expect this operation in real life? If the file IO is from your application, I would expect it not to do this without a good reason. And if the file IO is from another application (ie a restore by a backup application), then maybe regenerating multiple GB of data isn’t such a bad thing

Sent from Surface Pro


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTFSD is sponsored by OSR OSR is hiring!! Info at http://www.osr.com/careers For our schedule of debugging and file system seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTFSD is sponsored by OSR OSR is hiring!! Info at http://www.osr.com/careers For our schedule of debugging and file system seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx>

Yeah, it’s contains some ansi text with an Id stating where the file
originated

Something like this:
[ZoneTransfer]
ZoneId=3

The zones are listed here
http://msdn.microsoft.com/en-us/library/ie/ms537175(v=vs.85).aspx

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: 09 June 2014 05:18
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] file_overwrite deletes ads

I was surprised to see Microsoft?s Zone.Identifier streams disappear
too,

Is this the thing which causes ZIP files from email attachments to ask a
security question on open?

I was specifically thinking of security attributes when I mentioned read
only access, though the read only attribute does add another interesting set
of twists and turns. Gotta love writing filters :slight_smile:

The I/O Manager only makes these checks if the request is HANDLE based and
the requestor mode is User. The OP is bypassing these checks by using the
Flt APIs to perform the operations on the requestor’s behalf, hence my worry
about the lack of access checking.

-scott
OSR
@OSRDrivers

“Tony Mason” wrote in message news:xxxxx@ntfsd…

The read only bit is not a security attribute. It’s a file attribute.
Thus, as long as the caller can change the attributes
(FILE_WRITE_ATTRIBUTES), they can clear read-only. But neither the OS nor
the file system will NOT do it for them automatically. Changing the size of
a file (EOF, VDL, Allocation) requires FILE_WRITE_DATA. Changing the
attributes requires FILE_WRITE_ATTRIBUTES. This is enforced by the I/O
Manager, not the file system, so it’s not NTFS specific.

Supersede requires DELETE access, which isn’t impacted by the read-only bit.
Overwrite doesn’t require delete access and IS impacted by the read-only
bit.

On the flip side, the read only bit is not a security attribute. So even if
FILE_WRITE_ATTRIBUTES is granted as part of the file access, the file
systems will typically reject an open for write on the file. Thus,
read-only is enforced by the FSD, while the granted security (which is
associated with the given handle) is enforced by the I/O Manager.

Tangent: the read only bit is also a great way to screw up filter drivers.
I can create a new file, ask for FILE_WRITE_DATA access but set the
FILE_ATTRIBUTE_READONLY bit. The write access is granted, but now any
subsequent open of the file for write will fail - including an attempt to
write your ADS.

Tony
OSR

>free-form data. Secondly, the underlying implementation of ACLs and ADSs in NTFS are incredibly

different, so it was not just a matter of exposing the ACL implementation to the user.

Am I wrong that ACL is the ADS called $SecDesc?

What’s your definition of “classic notion of file”?

Byte stream of certain size, with the internal structure only understandable by the app layer.

For instance, how can you ZIP the file with EAs or ADSs? or Mac resource forks?

How can you attach such a file to email message?

How can you transfer it via the communication channel?

Are there any file systems that strictly adhere to the definition?

ext2/3/4

some subset of NTFS


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Surely adding metadata to the file data itself is a great idea. Look at OLE Documents, for instance.

SVN/GIT view of the world (as Maxim suggests) is great until people want to do something that
doesn’t conform to the SVN/GIT perspective.

Yes, but SVN/Git are much more capable then ADSs or Mac forks.

Like, oh, attaching the file to an e-mail…

This is a problem both with SVN/Git and with ADS too :slight_smile:

You should either have OLE-style internal container layout inside the file, or you should have companion files. Or you should have both as 2 choices, like the saved webpages by IE (.htm + a companion dir or .mht).

Since the e-mail program knows nothing about this other bit of meta-data, it gets lost. If it’s an
inherent part of the file, it doesn’t get lost.

ADSs are lost :slight_smile: they are not inherent part of the file stream, and email (MIME) can only attach streams.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

>attach a file. There are other apps which also ignore ADS within a file, dropbox being an example.

Surely, because DropBox have not originated on Windows/NTFS.

Additionally, moving a file to a USB stick (FAT32) leaves no choice but to ignore the stream as it’s
unsupported.

Note that both a) compound files b) companion files have no such issues.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com