For a test filter I am writing some data to an NTFS alternate data
stream (a type of shadow copy with some metadata appended).
I am having an issue when Explorer deletes such files. Namely, I am
NOT getting a Cleanup (or Close obviously) for those files 
I cleanup
my FltCreateFile handle to the ADS during Close, so this looks like
Catch 22.
If I release my references during delete
(Set_FileDispositionInformation), the files get deleted correctly, and I
do get the cleanup. However, this will not suffice, as I need the handle
as long as the file remains open, and is completely deleted (a Delete
call can be undone).
During IRP_MJ_CREATE, I call FltCreateFile on the ADS and keep the
handle in a stream context. I have removed any additional code apart
from FltCreateFile/FltClose (and ObRef/ObDerefObject on the handle to
get the File object), and the above still happens.
Is the ADS treated just like it were a main file, and a Cleanup
isn’t sent for the main file until its handles are closed? Or is this a
FltMgr thing? (yes, I consider it could be my fault, which is why I
removed all other code, to see if that’s the case)
XP, W7 - x86 and x64. Ideas are welcome.
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
Khm, a typo there. I AM getting a IRP_MJ_CLEANUP, but I am not getting a
IRP_MJ_CLOSE and a stream context cleanup callback.
Yes, I do need the handle between IRP_MJ_CLEANUP and IRP_MJ_CLOSE.
Dejan Maksimovic wrote:
For a test filter I am writing some data to an NTFS alternate data
stream (a type of shadow copy with some metadata appended).
I am having an issue when Explorer deletes such files. Namely, I am
NOT getting a Cleanup (or Close obviously) for those files I cleanup
my FltCreateFile handle to the ADS during Close, so this looks like
Catch 22.
If I release my references during delete
(Set_FileDispositionInformation), the files get deleted correctly, and I
do get the cleanup. However, this will not suffice, as I need the handle
as long as the file remains open, and is completely deleted (a Delete
call can be undone).
During IRP_MJ_CREATE, I call FltCreateFile on the ADS and keep the
handle in a stream context. I have removed any additional code apart
from FltCreateFile/FltClose (and ObRef/ObDerefObject on the handle to
get the File object), and the above still happens.
Is the ADS treated just like it were a main file, and a Cleanup
isn’t sent for the main file until its handles are closed? Or is this a
FltMgr thing? (yes, I consider it could be my fault, which is why I
removed all other code, to see if that’s the case)
XP, W7 - x86 and x64. Ideas are welcome.
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
Just so’s I’ve got this straight, the flow is
- Application opens “foo”
- in precreate you open “foo:bar”
- Application marks “foo” for delete. You see this
- Application closes “foo”. You see CLEANUP but not close.
- In close, (if you had seen it), you would have closed “foo:bar”.
I have no ides, but its not beyond the bounds of possibility that NTFS is
hanging on to the FO for “foo” so it can do something in that context.
- What happens if the application opens"foo:fred" (i.e. NTFS never sees an
open for “foo”)?
- For the purposes of experiment (because there’s a timing window), what
happens if you do the close in (4) if the file is marked for delete (and
opencount is zero &c &c)
- Equally, can you make sure that Cc doesn’t have its mits on anything?
(checkout the SOP)
“Dejan Maksimovic” wrote in message news:xxxxx@ntfsd…
Khm, a typo there. I AM getting a IRP_MJ_CLEANUP, but I am not getting
a
IRP_MJ_CLOSE and a stream context cleanup callback.
Yes, I do need the handle between IRP_MJ_CLEANUP and IRP_MJ_CLOSE.
Dejan Maksimovic wrote:
> For a test filter I am writing some data to an NTFS alternate data
> stream (a type of shadow copy with some metadata appended).
> I am having an issue when Explorer deletes such files. Namely, I am
> NOT getting a Cleanup (or Close obviously) for those files I cleanup
> my FltCreateFile handle to the ADS during Close, so this looks like
> Catch 22.
> If I release my references during delete
> (Set_FileDispositionInformation), the files get deleted correctly, and I
> do get the cleanup. However, this will not suffice, as I need the handle
> as long as the file remains open, and is completely deleted (a Delete
> call can be undone).
>
> During IRP_MJ_CREATE, I call FltCreateFile on the ADS and keep the
> handle in a stream context. I have removed any additional code apart
> from FltCreateFile/FltClose (and ObRef/ObDerefObject on the handle to
> get the File object), and the above still happens.
>
> Is the ADS treated just like it were a main file, and a Cleanup
> isn’t sent for the main file until its handles are closed? Or is this a
> FltMgr thing? (yes, I consider it could be my fault, which is why I
removed all other code, to see if that’s the case)
XP, W7 - x86 and x64. Ideas are welcome.
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
> Just so’s I’ve got this straight, the flow is
- Application opens “foo”
- in precreate you open “foo:bar”
- Application marks “foo” for delete. You see this
- Application closes “foo”. You see CLEANUP but not close.
- In close, (if you had seen it), you would have closed “foo:bar”.
Yep, that sums it up right.
- What happens if the application opens"foo:fred" (i.e. NTFS never sees an
open for “foo”)?
That one I didn’t try…
- For the purposes of experiment (because there’s a timing window), what
happens if you do the close in (4) if the file is marked for delete (and
opencount is zero &c &c)
Such a situation never occurs, I only see a Cleanup where the StreamCtx->UseCount is 2 (both my own counter
value and the one reported by !ctx extension).
If I only check whether it’s marked for deletion and close the handles, I do see a Close.
- Equally, can you make sure that Cc doesn’t have its mits on anything?
(checkout the SOP)
It does and here’s the strange part: I have seen cases where executable files get a CLOSE, but non
executables don’t (by executables I mean .exe only in this case - because Explorer opens them to get an Icon, but
files without a specific extension aren’t opened for it)
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
How mysterious. NTFS is a strange fish…
I guess another test would be to factor exploder out as well (so icon opens
and crap like that don’t happen). Which makes me thing, we are talking
about real delete, not move to the wastebasket, aren’t we…
R
Yes a true delete.
FYI, HandleEx is not showing an ADS open, but I guess it doesn’t show ADSes.
Rod Widdowson wrote:
How mysterious. NTFS is a strange fish…
I guess another test would be to factor exploder out as well (so icon opens
and crap like that don’t happen). Which makes me thing, we are talking
about real delete, not move to the wastebasket, aren’t we…
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
Dejan, it is your handle that is preventing the IRP_MJ_CLOSE from arriving. More accurately, it is allowing MM to retain references on the file. Delete happens on IRP_MJ_CLEANUP for the last handle to a file. As part of the delete cache sections get flushed and purged. This causes MM to release his references, resulting in the IRP_MJ_CLOSE.
Since you’re maintaining a handle, you’re preventing the delete, which in turn prevents the purge. MM will then keep his references to the file around for as long as he feels like keeping them, which will significantly delay the IRP_MJ_CLOSE.
– Christian [MSFT]
That would right, if I opened the main $DATA stream, but I am opening an alternate :MyTestStream:$DATA instead and keeping a handle to it. Why would MM keep a reference to the main $DATA stream then?
From what I can see in Fat code, non-paging FileDispositionInformation calls can be made even after IRP_MJ_CLEANUP went through, so I cannot release the reference during IRP_MJ_CLEANUP if the file is set for deletion (tried it, and the references ARE released every time, so the file gets deleted properly, but the above issue remains)
Dejan.
xxxxx@microsoft.com wrote:
Dejan, it is your handle that is preventing the IRP_MJ_CLOSE from arriving. More accurately, it is allowing MM to retain references on the file. Delete happens on IRP_MJ_CLEANUP for the last handle to a file. As part of the delete cache sections get flushed and purged. This causes MM to release his references, resulting in the IRP_MJ_CLOSE.
Since you’re maintaining a handle, you’re preventing the delete, which in turn prevents the purge. MM will then keep his references to the file around for as long as he feels like keeping them, which will significantly delay the IRP_MJ_CLOSE.
– Christian [MSFT]
NTFSD is sponsored by OSR
For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
Hi Christian,
So are you saying that having a handle open against any stream of a file will sometimes prevent IRP_MJ_CLOSE from arriving for FOs opened against the other streams of the same file ?
Thanks,
Alex.
Your handle to :MyTestStream:$DATA still counts as a handle to the file as a whole. NTFS doesn’t do any of the delete processing (including section purges) until the last handle to the entire file goes away. Since NTFS doesn’t issue the purge calls, MM doesn’t release his references to the default data stream’s section(s).
Perhaps I missed it, but what do you need that extra handle for?
– Christian [MSFT]
Just to be clear, the presence of other handles on other streams doesn’t *prevent* IRP_MJ_CLOSE from being sent to FOs for other streams. It simply doesn’t arrive in a timely fashion, based on the whims of CC.
– Christian [MSFT]
The alternate stream is used for some metadata on the main file (keeping track of changes).
xxxxx@microsoft.com wrote:
Your handle to :MyTestStream:$DATA still counts as a handle to the file as a whole. NTFS doesn’t do any of the delete processing (including section purges) until the last handle to the entire file goes away. Since NTFS doesn’t issue the purge calls, MM doesn’t release his references to the default data stream’s section(s).
Perhaps I missed it, but what do you need that extra handle for?
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
Very interesting, i didn’t know that…
Thanks Christian!
With that in mind, can I simply redirect paging I/O to the ADS to another file? (reads/setinfo calls). Something tells me this could be a synchronization issue. (tried it, but never in a production driver)
xxxxx@microsoft.com wrote:
Just to be clear, the presence of other handles on other streams doesn’t *prevent* IRP_MJ_CLOSE from being sent to FOs for other streams. It simply doesn’t arrive in a timely fashion, based on the whims of CC.
– Christian [MSFT]
NTFSD is sponsored by OSR
For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.
On 9/28/2011 2:33 PM, Dejan Maksimovic wrote:
With that in mind, can I simply redirect paging I/O to the ADS to another file? (reads/setinfo calls). Something tells me this could be a synchronization issue. (tried it, but never in a production driver)
Where is the I/O to the ADS coming from? I was under the impression
that your filter was the thing that had the handle open to the ADS and
that the filter was logging your change information to the ADS.
I was actually going to suggest using a separate change-tracking file so
that your filter could more easily avoid introducing complications with
delete.
–
Christian [MSFT]
This posting is provided “AS IS” with no warranties, and confers no rights.
> > With that in mind, can I simply redirect paging I/O to the ADS to another file? (reads/setinfo calls). Something tells me this could be a synchronization issue. (tried it, but never in a production driver)
Where is the I/O to the ADS coming from? I was under the impression that your filter was the thing that had the handle open to the ADS and that the filter was logging your change information to the ADS.
Yes, that is right. A read on the ADS can come from any application though.
I was actually going to suggest using a separate change-tracking file so that your filter could more easily avoid introducing complications with delete.
This is something we did before. Instead of opening the ADS (which would not work for any FS), we had a metadata folder. Unfortunately, there were several cases where an ADS would work where a metadata
storage would not. It’s because of those cases that I need to use an ADS.
–
Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.