File system driver. How to begin

OSR_Community_User · May 21, 2004, 9:11am

Hello.
I’m going to write virtual FS driver (without underlying device) to do the job similar to what Exchange server’s BackOfficeStorage (drive M:; driver exifs.sys) does (similar but not exactly the same, therefore the need for development). My question is about general relationsips between things: what should I do in my driver in order to have the same effect. exifs creates symbolic links (??\BackOfficeStorage points to \Device\ExchangeIfsDevice and M: points to ??\BackOfficeStorage; \Device\ExchangeIfsDevice is created by exifs).
Are the two links enough for the system to think that M: is filesystem and send to the driver all the file-related IRPs?
Also this driver (exifs) calls RxRegisterMinirdr so it seems to be network filesystem driver, but it doesn’ seem to have accompanying provider dll (I’ve compared it with nulmrx sample from DDK; provider dll means something registered in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\NetworkProvider\Order).

So, my questions are:

how to implement such a thing in general (FS driver without underlying block device) - network file system or…? Architectural outline.
if network file system - what is the role of “Network Provider”?

Jacob

OSR_Community_User · May 21, 2004, 9:40am

Here at OSR we refer to these as “pseudo file systems”. One of the more
interesting pseudo file systems we ever constructed was backed by the
bit-bucket: writes were discarded, reads zeroed the caller’s read
buffer, directory structure was maintained in memory. It was amazing
how long the data would still be there (in the cache, of course).

But we’ve seen all sorts of pseudo file systems of the type you
describe. The file systems interface that we discuss here in NTFSD is
mostly related to the upper edge (from the I/O Manager and other OS
components) and the side edge (into the cache and associated
interactions) but not very much about the lower edge (where the data
actually is stored).

Network redirectors need not provide an NP - that just means the
redirector in question doesn’t support dynamic drive connection, which
would explain the comments you made about the drive letter symbolic link
it creates - that is how apps find this drive. Using the
mini-redirector framework might ease your implementation (I’ve not done
it, so I can’t honestly say one way or the other.)

At one point (meaning I am not sure if it is still there), Microsoft
Research distributed a user mode file system - it was a “reflector”
model that took kernel mode requests and sent them to user mode. That
is a good model for pseudo file systems, but is horrible if you decide
you want to store anything that might be needed by Win32, since this
leads to interesting and bizarre deadlocks.

Good luck. If you continue down this road, we’ll no doubt be hearing
more from you!

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the Next OSR File Systems Class October
18, 2004 in Silicon Valley!

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jacob
Sent: Friday, May 21, 2004 9:10 AM
To: ntfsd redirect
Subject: [ntfsd] File system driver. How to begin

Hello.
I’m going to write virtual FS driver (without underlying device) to do
the job similar to what Exchange server’s BackOfficeStorage (drive M:;
driver exifs.sys) does (similar but not exactly the same, therefore the
need for development). My question is about general relationsips between
things: what should I do in my driver in order to have the same effect.
exifs creates symbolic links (??\BackOfficeStorage points to
\Device\ExchangeIfsDevice and M: points to ??\BackOfficeStorage;
\Device\ExchangeIfsDevice is created by exifs).
Are the two links enough for the system to think that M: is filesystem
and send to the driver all the file-related IRPs?
Also this driver (exifs) calls RxRegisterMinirdr so it seems to be
network filesystem driver, but it doesn’ seem to have accompanying
provider dll (I’ve compared it with nulmrx sample from DDK; provider dll
means something registered in
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\NetworkProvider\Orde
r).

So, my questions are:

how to implement such a thing in general (FS driver without
underlying block device) - network file system or…? Architectural
outline.
if network file system - what is the role of “Network Provider”?

Jacob

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Prokash_Sinha-1 · May 21, 2004, 9:45am

I know several person(s) can write essays on it, and I’m not the one to say
anything about it … :).

Rajeev Nagar’s whole book is devoted to this. He is definitely one, among
others to shed some lights on it

-pro

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Jacob
Sent: Friday, May 21, 2004 6:10 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] File system driver. How to begin

Hello.
I’m going to write virtual FS driver (without underlying device) to do the
job similar to what Exchange server’s BackOfficeStorage (drive M:; driver
exifs.sys) does (similar but not exactly the same, therefore the need for
development). My question is about general relationsips between things: what
should I do in my driver in order to have the same effect. exifs creates
symbolic links (??\BackOfficeStorage points to \Device\ExchangeIfsDevice
and M: points to ??\BackOfficeStorage; \Device\ExchangeIfsDevice is created
by exifs).
Are the two links enough for the system to think that M: is filesystem and
send to the driver all the file-related IRPs?
Also this driver (exifs) calls RxRegisterMinirdr so it seems to be network
filesystem driver, but it doesn’ seem to have accompanying provider dll
(I’ve compared it with nulmrx sample from DDK; provider dll means something
registered in
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\NetworkProvider\Order).

So, my questions are:

how to implement such a thing in general (FS driver without underlying
block device) - network file system or…? Architectural outline.
if network file system - what is the role of “Network Provider”?

Jacob

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@garlic.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

OSR_Community_User · May 21, 2004, 5:46pm

Tony,
Thank you for response.

You wrote:
The file systems interface that we discuss here in NTFSD is
mostly related to the upper edge (from the I/O Manager and other OS
components) and the side edge (into the cache and associated
interactions) but not very much about the lower edge (where the data
actually is stored).

That is exactly my question: how to make FS driver that appears to the upper
edge as normal filesystem with drive letter, but where to get data will be
my own business (beyond this discussion). I have such an example for Windows
95 - regfsd from NuMega(?) - expose system registry as file system. How to
do similar things in Windows 2000/2003?

You wrote:
Network redirectors need not provide an NP - that just means the
redirector in question doesn’t support dynamic drive connection, which
would explain the comments you made about the drive letter symbolic link
it creates - that is how apps find this drive. Using the
mini-redirector framework might ease your implementation (I’ve not done
it, so I can’t honestly say one way or the other.)

So, I should write it as network redirector, create the “M:” symbolic link
to it and all will be OK, right? (OK means here that upper edge will happily
acept this M: as disk drive and will send appropriate IRPs and I will only
have to satisfy them one way or another)?

You wrote:
At one point (meaning I am not sure if it is still there), Microsoft
Research distributed a user mode file system - it was a “reflector”
model that took kernel mode requests and sent them to user mode. That
is a good model for pseudo file systems, but is horrible if you decide
you want to store anything that might be needed by Win32, since this
leads to interesting and bizarre deadlocks.

I have found this link at Bo Branten site (dead, but its parent is alive
(personal page) and contains Usenix article about that). I don’t remember
the URL, I’ve seen it at work, but now I’m at home. I’ve read it and
probably will have to do smth like (but not as general) to extract
information from Exchange store. That will be another story. First I’d like
to establish framework (working model of Filesystem driver)

Good luck. If you continue down this road, we’ll no doubt be hearing
more from you!

Why not?

Regards,

Jacob

OSR_Community_User · May 22, 2004, 8:51pm

> Network redirectors need not provide an NP - that just means the

Not providing an NP with a redir will lead to a load of problems with
representing the drive in the shell.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

OSR_Community_User · May 23, 2004, 5:29am

I disagree. A network redirector that utilizes a fixed drive letter
(e.g., AFS or DCE/DFS) need not provide a network provider library.
We’ve done this with other redirectors as well and have not seen
“problems with representing the drive in the shell”. The shell sees the
drives without problem, provides access, etc. - but via the drive
letter, not via the MPR.

Actually, I think that having an NP on a network redirector that does
not support UNC would be a true problem and lead to browsing issues of
the type that you describe.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Saturday, May 22, 2004 8:53 PM
To: ntfsd redirect
Subject: Re: Re:[ntfsd] File system driver. How to begin

Network redirectors need not provide an NP - that just means the

Not providing an NP with a redir will lead to a load of problems with
representing the drive in the shell.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com To unsubscribe
send a blank email to xxxxx@lists.osr.com

OSR_Community_User · May 23, 2004, 2:39pm

Strange. For me, if there is no NP DLL, then the redir’s drive leller is
always represented as “Disconnected Network Drive” in the shell.

Maybe I should not claim the redir to be a network drive in its DeviceType,
but for now it claims to be a network file system. Network file systems need NP
DLL to be represented in the shell.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Tony Mason”
To: “Windows File Systems Devs Interest List”
Sent: Sunday, May 23, 2004 1:31 PM
Subject: RE: Re:[ntfsd] File system driver. How to begin

> I disagree. A network redirector that utilizes a fixed drive letter
> (e.g., AFS or DCE/DFS) need not provide a network provider library.
> We’ve done this with other redirectors as well and have not seen
> “problems with representing the drive in the shell”. The shell sees the
> drives without problem, provides access, etc. - but via the drive
> letter, not via the MPR.
>
> Actually, I think that having an NP on a network redirector that does
> not support UNC would be a true problem and lead to browsing issues of
> the type that you describe.
>
> Regards,
>
> Tony
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
> Sent: Saturday, May 22, 2004 8:53 PM
> To: ntfsd redirect
> Subject: Re: Re:[ntfsd] File system driver. How to begin
>
> > Network redirectors need not provide an NP - that just means the
>
> Not providing an NP with a redir will lead to a load of problems with
> representing the drive in the shell.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@osr.com To unsubscribe
> send a blank email to xxxxx@lists.osr.com
>
>
>
> —
> Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

OSR_Community_User · June 1, 2004, 6:38pm

Thank you for help.

“Maxim S. Shatskih” ???/??? ? ???
???: news:xxxxx@ntfsd…
> Strange. For me, if there is no NP DLL, then the redir’s drive leller
is
> always represented as “Disconnected Network Drive” in the shell.
>
> Maybe I should not claim the redir to be a network drive in its
DeviceType,

Probably. Exchange’s drive is shown in Explorer among hard disks rather than
among network connections. Its size is the same as “real” hard disk size. By
the way it raises interesting question: what is DeviceType? The
documentation says absolutely nothing about it - only list of posiible
values. But what is its role in the system, what are the consequences of
declaring DeviceType this or that - no information.

I followed Tony’s advice and have been trying to make mini redirector (as
exifs does), but I have lots of troubles now. There are some cryptic objects
(NetRoot, FOBX), no documentation… So, the question remains: what is the
best way: redirector or simple driver (based on Rajeev Nagar’s sample, for
example). In other words, what are the benefits of being redirector?

Jacob

> but for now it claims to be a network file system. Network file systems
need NP
> DLL to be represented in the shell.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> ----- Original Message -----
> From: “Tony Mason”
> To: “Windows File Systems Devs Interest List”
> Sent: Sunday, May 23, 2004 1:31 PM
> Subject: RE: Re:[ntfsd] File system driver. How to begin
>
>
> > I disagree. A network redirector that utilizes a fixed drive letter
> > (e.g., AFS or DCE/DFS) need not provide a network provider library.
> > We’ve done this with other redirectors as well and have not seen
> > “problems with representing the drive in the shell”. The shell sees the
> > drives without problem, provides access, etc. - but via the drive
> > letter, not via the MPR.
> >
> > Actually, I think that having an NP on a network redirector that does
> > not support UNC would be a true problem and lead to browsing issues of
> > the type that you describe.
> >
> > Regards,
> >
> > Tony
> >
> > Tony Mason
> > Consulting Partner
> > OSR Open Systems Resources, Inc.
> > http://www.osr.com
> >
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
> > Sent: Saturday, May 22, 2004 8:53 PM
> > To: ntfsd redirect
> > Subject: Re: Re:[ntfsd] File system driver. How to begin
> >
> > > Network redirectors need not provide an NP - that just means the
> >
> > Not providing an NP with a redir will lead to a load of problems with
> > representing the drive in the shell.
> >
> > Maxim Shatskih, Windows DDK MVP
> > StorageCraft Corporation
> > xxxxx@storagecraft.com
> > http://www.storagecraft.com
> >
> >
> > —
> > Questions? First check the IFS FAQ at
> > https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@osr.com To unsubscribe
> > send a blank email to xxxxx@lists.osr.com
> >
> >
> >
> > —
> > Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
>
>

OSR_Community_User · June 2, 2004, 10:25am

> Probably. Exchange’s drive is shown in Explorer among hard disks rather than

among network connections. Its size is the same as “real” hard disk size. By

Is Exchange’s drive implemented by shell extension or by real FSD?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

OSR_Community_User · June 2, 2004, 1:15pm

----- Original Message -----
From: “Maxim S. Shatskih”
To: “Windows File Systems Devs Interest List”
Sent: Wednesday, June 02, 2004 6:26 PM
Subject: Re: Re:[ntfsd] Re:File system driver. How to begin

> > Probably. Exchange’s drive is shown in Explorer among hard disks rather
than
> > among network connections. Its size is the same as “real” hard disk
size. By
>
> Is Exchange’s drive implemented by shell extension or by real FSD?
By real FSD (driver exifs.sys). See my initial post for details (of May 21).
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@umail.ru
> To unsubscribe send a blank email to xxxxx@lists.osr.com

Lyndon_J_Clarke-2 · June 6, 2004, 10:23am

I think it would be fairest to say its a pseudo-filesystem. There is a
driver exifs.sys which provides \FileSystem\EXIFS however it uses NTFS for
storage.

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntfsd…
> > Probably. Exchange’s drive is shown in Explorer among hard disks rather
than
> > among network connections. Its size is the same as “real” hard disk
size. By
>
> Is Exchange’s drive implemented by shell extension or by real FSD?
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>