Re[4]: Tracking file objects

Ged,

While I agree with you that it is a possibility it is basically how NTFS
and FAT perform validation on the file objects they own. Try passing
down one of your file objects to NTFS and the system will crash. Analyze
where it crashes and you will see the checks it is performing …

This is not saying that it is a way to ensure 100% consistent behavior
nor am I saying that “if Microsoft does it then it must be right” but I
am saying that in the decades of implementing layered file systems, I
have never run into a collision … that I know of.

Pete


Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295

------ Original Message ------
From: “Ged Murphy”
To: “Windows File Systems Devs Interest List”
Sent: 1/22/2015 8:10:02 AM
Subject: RE: Re[2]: [ntfsd] Tracking file objects

>Yeah I thought about this route initially, but then how unique is a
>unique signature? I realise it’s a million-to-one chance, but it’s
>possible that some other filter (or 3rd party file system) uses
>FsContext2 and the first 4 bytes may match up at some point.
>
>Considering the file object address is guaranteed to be unique, isn’t
>that a better option?
>
>
>-----Original Message-----
>From: xxxxx@lists.osr.com
>[mailto:xxxxx@lists.osr.com] On Behalf Of PScott
>Sent: 22 January 2015 14:59
>To: Windows File Systems Devs Interest List
>Subject: Re[2]: [ntfsd] Tracking file objects
>
>
>Ged,
>
>For the upper file objects which you own, you have setup the FsContext
>and FsContext2 pointers to point to your own control structures. What
>you can do is setup the first 4 bytes of the FsContext2 structure to
>point to a unique signature for your file objects. This way, perform
>validation on the pointer and then evaluate the first 4 bytes to
>determine if it is your file object or not.
>
>Pete
>
>–
>Kernel Drivers
>Windows File System and Device Driver Consulting
>www.KernelDrivers.com
>866.263.9295
>
>
>
>
>------ Original Message ------
>From: “Ged Murphy”
>To: “Windows File Systems Devs Interest List”
>Sent: 1/22/2015 5:26:02 AM
>Subject: RE: [ntfsd] Tracking file objects
>
>>Yeah I have a similar setup, the lower file objects are no problem.
>>What I’m referring to is in the isolation model, we obviously have to
>>ensure that the FO’s we own never go below our filter. So how do we
>>determine which upper file objects are ones we own, and which are one
>>that the FSD owns.
>>
>>In normal (non-isolation) filters, most people make a decision in the
>>create callback and set a streamhandle context for the FO. They can
>>then use FltGetStreamHandleContext in all the other callbacks to check
>>if it’s a FO they’re interested in.
>>
>>What I’m wondering is In the isolation model do people do things
>>differently? Is it preferred to just keep a linked list of all the
>>upper FO’s we own and check this list for every callback, or do I just
>>use FltSetStreamHandleContext in the create callback on my own FO and
>>check it as usual in all the others.
>>
>>I realise I’m probable overthinking this but it’s a question I keep
>>asking myself and haven’t come to a decision on ‘best practice’
>>
>>Ged.
>>
>>
>>-----Original Message-----
>>From: xxxxx@lists.osr.com
>>[mailto:xxxxx@lists.osr.com] On Behalf Of Rod Widdowson
>>Sent: 22 January 2015 12:06
>>To: Windows File Systems Devs Interest List
>>Subject: Re:[ntfsd] Tracking file objects
>>
>>> When implementing the isolation model, what’s the preferred way of
>>> tracking file objects that we own?
>>
>>In any filter I usually find it handy to have my StreamHandleContexts
>>in a linked list off my StreamContexts and my StreamContexts off my
>>VolumeContext. This should give you all you need to be able to find
>>all
>>the lower file objects (since your lower file objects are in your
>>StreamHandleContext, right).
>>
>>Unrelated to this but when you are working out your navigation you
>>need
>>to bear in mind the exigies of oplocks which require you to be able to
>>traverse all the children (to break an oplock to allow a directory
>>rename to work).
>>ISTR that there are also cases when you need to be able to traverse
>>upwards as well.
>>
>>
>>
>>—
>>NTFSD is sponsored by OSR
>>
>>OSR is hiring!! Info at http://www.osr.com/careers
>>
>>For our schedule of debugging and file system seminars visit:
>>http://www.osr.com/seminars
>>
>>To unsubscribe, visit the List Server section of OSR Online at
>>http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>>—
>>NTFSD is sponsored by OSR
>>
>>OSR is hiring!! Info at http://www.osr.com/careers
>>
>>For our schedule of debugging and file system seminars visit:
>>http://www.osr.com/seminars
>>
>>To unsubscribe, visit the List Server section of OSR Online at
>>http://www.osronline.com/page.cfm?name=ListServer
>
>
>—
>NTFSD is sponsored by OSR
>
>OSR is hiring!! Info at http://www.osr.com/careers
>
>For our schedule of debugging and file system seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer
>
>
>—
>NTFSD is sponsored by OSR
>
>OSR is hiring!! Info at http://www.osr.com/careers
>
>For our schedule of debugging and file system seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer

Ahh so it does. Looking at the fastfat souce now, it checks for
*Ccb == FAT_NTC_CCB
Which is a 4 byte value
#define FAT_NTC_CCB ((NODE_TYPE_CODE)0x0507)

I’ve also just dumped a few NTFS file objects and their FsContext2 all have a 4 byte value of 0x0709.

Thanks for the info Pete, I’ll follow your advice and switch to a unique number (other than those two…).
If it’s good enough for MS then it’s good enough for me :slight_smile:

Ged.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of PScott
Sent: 22 January 2015 16:33
To: Windows File Systems Devs Interest List
Subject: Re[4]: [ntfsd] Tracking file objects

Ged,

While I agree with you that it is a possibility it is basically how NTFS and FAT perform validation on the file objects they own. Try passing down one of your file objects to NTFS and the system will crash. Analyze where it crashes and you will see the checks it is performing …

This is not saying that it is a way to ensure 100% consistent behavior nor am I saying that “if Microsoft does it then it must be right” but I am saying that in the decades of implementing layered file systems, I have never run into a collision … that I know of.

Pete


Kernel Drivers
Windows File System and Device Driver Consulting www.KernelDrivers.com
866.263.9295

------ Original Message ------
From: “Ged Murphy”
To: “Windows File Systems Devs Interest List”
Sent: 1/22/2015 8:10:02 AM
Subject: RE: Re[2]: [ntfsd] Tracking file objects

>Yeah I thought about this route initially, but then how unique is a
>unique signature? I realise it’s a million-to-one chance, but it’s
>possible that some other filter (or 3rd party file system) uses
>FsContext2 and the first 4 bytes may match up at some point.
>
>Considering the file object address is guaranteed to be unique, isn’t
>that a better option?
>
>
>-----Original Message-----
>From: xxxxx@lists.osr.com
>[mailto:xxxxx@lists.osr.com] On Behalf Of PScott
>Sent: 22 January 2015 14:59
>To: Windows File Systems Devs Interest List
>Subject: Re[2]: [ntfsd] Tracking file objects
>
>
>Ged,
>
>For the upper file objects which you own, you have setup the FsContext
>and FsContext2 pointers to point to your own control structures. What
>you can do is setup the first 4 bytes of the FsContext2 structure to
>point to a unique signature for your file objects. This way, perform
>validation on the pointer and then evaluate the first 4 bytes to
>determine if it is your file object or not.
>
>Pete
>
>–
>Kernel Drivers
>Windows File System and Device Driver Consulting www.KernelDrivers.com
>866.263.9295
>
>
>
>
>------ Original Message ------
>From: “Ged Murphy”
>To: “Windows File Systems Devs Interest List”
>Sent: 1/22/2015 5:26:02 AM
>Subject: RE: [ntfsd] Tracking file objects
>
>>Yeah I have a similar setup, the lower file objects are no problem.
>>What I’m referring to is in the isolation model, we obviously have to
>>ensure that the FO’s we own never go below our filter. So how do we
>>determine which upper file objects are ones we own, and which are one
>>that the FSD owns.
>>
>>In normal (non-isolation) filters, most people make a decision in the
>>create callback and set a streamhandle context for the FO. They can
>>then use FltGetStreamHandleContext in all the other callbacks to check
>>if it’s a FO they’re interested in.
>>
>>What I’m wondering is In the isolation model do people do things
>>differently? Is it preferred to just keep a linked list of all the
>>upper FO’s we own and check this list for every callback, or do I just
>>use FltSetStreamHandleContext in the create callback on my own FO and
>>check it as usual in all the others.
>>
>>I realise I’m probable overthinking this but it’s a question I keep
>>asking myself and haven’t come to a decision on ‘best practice’
>>
>>Ged.
>>
>>
>>-----Original Message-----
>>From: xxxxx@lists.osr.com
>>[mailto:xxxxx@lists.osr.com] On Behalf Of Rod Widdowson
>>Sent: 22 January 2015 12:06
>>To: Windows File Systems Devs Interest List
>>Subject: Re:[ntfsd] Tracking file objects
>>
>>> When implementing the isolation model, what’s the preferred way of
>>> tracking file objects that we own?
>>
>>In any filter I usually find it handy to have my StreamHandleContexts
>>in a linked list off my StreamContexts and my StreamContexts off my
>>VolumeContext. This should give you all you need to be able to find
>>all the lower file objects (since your lower file objects are in your
>>StreamHandleContext, right).
>>
>>Unrelated to this but when you are working out your navigation you
>>need to bear in mind the exigies of oplocks which require you to be
>>able to traverse all the children (to break an oplock to allow a
>>directory rename to work).
>>ISTR that there are also cases when you need to be able to traverse
>>upwards as well.
>>
>>
>>
>>—
>>NTFSD is sponsored by OSR
>>
>>OSR is hiring!! Info at http://www.osr.com/careers
>>
>>For our schedule of debugging and file system seminars visit:
>>http://www.osr.com/seminars
>>
>>To unsubscribe, visit the List Server section of OSR Online at
>>http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>>—
>>NTFSD is sponsored by OSR
>>
>>OSR is hiring!! Info at http://www.osr.com/careers
>>
>>For our schedule of debugging and file system seminars visit:
>>http://www.osr.com/seminars
>>
>>To unsubscribe, visit the List Server section of OSR Online at
>>http://www.osronline.com/page.cfm?name=ListServer
>
>
>—
>NTFSD is sponsored by OSR
>
>OSR is hiring!! Info at http://www.osr.com/careers
>
>For our schedule of debugging and file system seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer
>
>
>—
>NTFSD is sponsored by OSR
>
>OSR is hiring!! Info at http://www.osr.com/careers
>
>For our schedule of debugging and file system seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Although thinking about this (after I unfortunately clicked send in my last mail), due to the IoManager, the NTFS and FAT driver should only ever receive the FO’s that belong to them, so the validation they perform is fine.

In the case of an isolation filter, we receive all file objects and we’re doing the actual identification normally reserved for the IoManager, not just validation.

-----Original Message-----
From: Ged Murphy [mailto:xxxxx@gmail.com]
Sent: 22 January 2015 17:12
To: ‘Windows File Systems Devs Interest List’
Subject: RE: Re[4]: [ntfsd] Tracking file objects

Ahh so it does. Looking at the fastfat souce now, it checks for
*Ccb == FAT_NTC_CCB
Which is a 4 byte value
#define FAT_NTC_CCB ((NODE_TYPE_CODE)0x0507)

I’ve also just dumped a few NTFS file objects and their FsContext2 all have a 4 byte value of 0x0709.

Thanks for the info Pete, I’ll follow your advice and switch to a unique number (other than those two…).
If it’s good enough for MS then it’s good enough for me :slight_smile:

Ged.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of PScott
Sent: 22 January 2015 16:33
To: Windows File Systems Devs Interest List
Subject: Re[4]: [ntfsd] Tracking file objects

Ged,

While I agree with you that it is a possibility it is basically how NTFS and FAT perform validation on the file objects they own. Try passing down one of your file objects to NTFS and the system will crash. Analyze where it crashes and you will see the checks it is performing …

This is not saying that it is a way to ensure 100% consistent behavior nor am I saying that “if Microsoft does it then it must be right” but I am saying that in the decades of implementing layered file systems, I have never run into a collision … that I know of.

Pete


Kernel Drivers
Windows File System and Device Driver Consulting www.KernelDrivers.com
866.263.9295

------ Original Message ------
From: “Ged Murphy”
To: “Windows File Systems Devs Interest List”
Sent: 1/22/2015 8:10:02 AM
Subject: RE: Re[2]: [ntfsd] Tracking file objects

>Yeah I thought about this route initially, but then how unique is a
>unique signature? I realise it’s a million-to-one chance, but it’s
>possible that some other filter (or 3rd party file system) uses
>FsContext2 and the first 4 bytes may match up at some point.
>
>Considering the file object address is guaranteed to be unique, isn’t
>that a better option?
>
>
>-----Original Message-----
>From: xxxxx@lists.osr.com
>[mailto:xxxxx@lists.osr.com] On Behalf Of PScott
>Sent: 22 January 2015 14:59
>To: Windows File Systems Devs Interest List
>Subject: Re[2]: [ntfsd] Tracking file objects
>
>
>Ged,
>
>For the upper file objects which you own, you have setup the FsContext
>and FsContext2 pointers to point to your own control structures. What
>you can do is setup the first 4 bytes of the FsContext2 structure to
>point to a unique signature for your file objects. This way, perform
>validation on the pointer and then evaluate the first 4 bytes to
>determine if it is your file object or not.
>
>Pete
>
>–
>Kernel Drivers
>Windows File System and Device Driver Consulting www.KernelDrivers.com
>866.263.9295
>
>
>
>
>------ Original Message ------
>From: “Ged Murphy”
>To: “Windows File Systems Devs Interest List”
>Sent: 1/22/2015 5:26:02 AM
>Subject: RE: [ntfsd] Tracking file objects
>
>>Yeah I have a similar setup, the lower file objects are no problem.
>>What I’m referring to is in the isolation model, we obviously have to
>>ensure that the FO’s we own never go below our filter. So how do we
>>determine which upper file objects are ones we own, and which are one
>>that the FSD owns.
>>
>>In normal (non-isolation) filters, most people make a decision in the
>>create callback and set a streamhandle context for the FO. They can
>>then use FltGetStreamHandleContext in all the other callbacks to check
>>if it’s a FO they’re interested in.
>>
>>What I’m wondering is In the isolation model do people do things
>>differently? Is it preferred to just keep a linked list of all the
>>upper FO’s we own and check this list for every callback, or do I just
>>use FltSetStreamHandleContext in the create callback on my own FO and
>>check it as usual in all the others.
>>
>>I realise I’m probable overthinking this but it’s a question I keep
>>asking myself and haven’t come to a decision on ‘best practice’
>>
>>Ged.
>>
>>
>>-----Original Message-----
>>From: xxxxx@lists.osr.com
>>[mailto:xxxxx@lists.osr.com] On Behalf Of Rod Widdowson
>>Sent: 22 January 2015 12:06
>>To: Windows File Systems Devs Interest List
>>Subject: Re:[ntfsd] Tracking file objects
>>
>>> When implementing the isolation model, what’s the preferred way of
>>> tracking file objects that we own?
>>
>>In any filter I usually find it handy to have my StreamHandleContexts
>>in a linked list off my StreamContexts and my StreamContexts off my
>>VolumeContext. This should give you all you need to be able to find
>>all the lower file objects (since your lower file objects are in your
>>StreamHandleContext, right).
>>
>>Unrelated to this but when you are working out your navigation you
>>need to bear in mind the exigies of oplocks which require you to be
>>able to traverse all the children (to break an oplock to allow a
>>directory rename to work).
>>ISTR that there are also cases when you need to be able to traverse
>>upwards as well.
>>
>>
>>
>>—
>>NTFSD is sponsored by OSR
>>
>>OSR is hiring!! Info at http://www.osr.com/careers
>>
>>For our schedule of debugging and file system seminars visit:
>>http://www.osr.com/seminars
>>
>>To unsubscribe, visit the List Server section of OSR Online at
>>http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>>—
>>NTFSD is sponsored by OSR
>>
>>OSR is hiring!! Info at http://www.osr.com/careers
>>
>>For our schedule of debugging and file system seminars visit:
>>http://www.osr.com/seminars
>>
>>To unsubscribe, visit the List Server section of OSR Online at
>>http://www.osronline.com/page.cfm?name=ListServer
>
>
>—
>NTFSD is sponsored by OSR
>
>OSR is hiring!! Info at http://www.osr.com/careers
>
>For our schedule of debugging and file system seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer
>
>
>—
>NTFSD is sponsored by OSR
>
>OSR is hiring!! Info at http://www.osr.com/careers
>
>For our schedule of debugging and file system seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

I must admit I have never relied on magick numbers. I’m more brutal, I just
make my Ccb (FsContext2) a StreamHandleContext on the upper file object.

in in PreCall you get the StreamHandleContext

  • If there isn’t one its not your file object
  • If it there is you can inspect it to see whether it has a lower file
    object.

By contrast I make the Scb (FsContext) a StreamContext on the Lower File
Object.

So you’re saying in the PreCreate:

  • Create the lower FO (shadow file)
  • Copy the bits up to the upper FO
  • Create and initialize FsContext & FsContext2 for the upper FO
  • Call FltSetStreamHandleContext on the upper FO, and store the address of FsContext2 you just created
  • Release

On all other calls:

  • Call FltGetStreamHandleContext on the upper FO to check if it’s one you own

That was one of the original ideas I’ve been toying around with, the other option was a linked list of all the FO’s I own, and traverse the list on every call (instead of calling FltGetStreamHandleContext)

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Rod Widdowson
Sent: 22 January 2015 17:47
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Tracking file objects

I must admit I have never relied on magick numbers. I’m more brutal, I just make my Ccb (FsContext2) a StreamHandleContext on the upper file object.

in in PreCall you get the StreamHandleContext

  • If there isn’t one its not your file object
  • If it there is you can inspect it to see whether it has a lower file object.

By contrast I make the Scb (FsContext) a StreamContext on the Lower File Object.


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer