I need to develop a file system based ntfs on win2k.But I can’t find any example. Where Can I find any suggestions?
Thanks.
I would like the winning lottery ticket that was sold in Ohio too. I don’t
expect to get it and you shouldn’t either. Developing a FSD is a major task
that very few can accomplish. OSR sells a development kit that have
samples.
“like” wrote in message news:xxxxx@ntfsd…
>
> I need to develop a file system based ntfs on win2k.But I can’t find any
> example. Where Can I find any suggestions?
> Thanks.
>
>
>
>
Thanks for your suggestion.
I must accomplish the task because it is my job that my boss give me.
I would like the winning lottery ticket that was sold in Ohio too. I don’t
expect to get it and you shouldn’t either. Developing a FSD is a major task
that very few can accomplish. OSR sells a development kit that have
samples.“like” wrote in message news:xxxxx@ntfsd…
>>
>> I need to develop a file system based ntfs on win2k.But I can’t find any
>> example. Where Can I find any suggestions?
>> Thanks.
>>
>>
>>
>>
>
>
>
>—
>Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17
>
>You are currently subscribed to ntfsd as: likeosr@126.com
>To unsubscribe send a blank email to xxxxx@lists.osr.com
>
= = = = = = = = = = = = = = = = = = = =
I think David’s point is that what you are asking about is an extremely
challenging task to accomplish. You can start with the IFS Kit (or the
Windows Vista WDK which subsumes the Windows Server 2003 SP 1 IFS Kit)
and examine the samples that it contains. Depending upon the
requirements of your project, you will find that a project to implement
a functional file system (similar to FAT, for example) would likely be
2-3 years of time for a single developer, particularly one who has to
learn about Windows file systems. Someone who has experience developing
Windows file systems can likely accomplish the task in around 1 year.
As you increase the requirements and complexity of the project, you will
quickly see that time estimate increase (one project on which I worked
in the past included 125 engineering years of effort for the Version 1.0
product.)
This may not be in line with your boss’s expectations. If that is the
case, I would suggest that you educate your boss, find a different boss,
or prepare to have him blame you when you are not able to meet his
(unrealistic) expectations. I know it is often difficult to deliver bad
news to a boss, but it is far better to educate him now than it will be
to have him think that you are incompetent when you must struggle to
understand the rather complex file systems model inside Windows. If,
for instance, your boss believes this is comparable to developing a
Linux file system, your boss would be incorrect - it is considerably
more complex, in no small part because of the file systems model that
has been adopted by the Windows team.
Assuming that your boss wants you to develop a file system “from
scratch” and does not consider the cost of your time in developing it to
have any value, my suggestion would be to obtain a copy of the Windows
Server 2003 SP1 IFS Kit (from Microsoft, US $109 plus shipping and
handling) and a copy of the Windows NT File Systems Development book
(which OSR offers in a reprint version available on the OSR Online
store, or you can purchase in the used book market.) If your boss is
unwilling to expend the 2-3 year effort for you to obtain on-the-job
training, I would suggest that you consider taking the file systems
class (I’m teaching again in April) next month. Generally, I’ve been
told by people who have taken the class that it has helped their
development process immensely (I have several people who have taken it
more than once - there is a 50% discount to take the class a second time
- and who have told me that each time they have learned more from the
class.)
If your boss is more concerned about market pressures (e.g. “time to
market” and “total cost of ownership”) then he might be better off
considering working with any one of the consultants that work in this
space. While OSR does work in this space, we typically only engage in
complex technical projects and from your brief description it does not
sound like your project is technically complex. In addition there are
several other people who can also work with you, several of whom
participate in this forum.
On the positive side for you, the skills to develop a file system are
valuable. There are few experienced developers, and only a handful in
the world who have been involved in multiple file system development
projects. My experience is that companies often MUST train their own
staff because they cannot find anyone with these specialized skills.
File systems is a challenging and complex area in which to work. Unlike
application programming (where there is often a concept of “close
enough”) it is intolerant of errors. I usually compare it to the
“payroll department” in your company - people do not tolerate errors in
the payroll process (“oh, gee we forgot to pay you this month. No
worries, we’ll catch up next month” is similar to “oh, gee we lost your
data. No worries, we’ll keep the next file you give to us”.) And
almost all of the really interesting parts of file systems development
relate to handling failures - failures in the network, the storage
system, this computer, other computers. Byzantine failures are
particularly “fun” - at FAST '05 Jim Gray (Microsoft Research) talked
about how the Exchange team was noticing a class of disk errors that
neither the NTFS nor the SQL team observed. It turns out that Exchange
implements an end-to-end checksum on the data that they write to disk,
while NTFS and SQL do not. As a result, Exchange was noticing errors
where the data returned from the disk was not what was written to the
disk. (This is increasingly common, as disk drive densities become
higher and higher there are issues related to adjacent bits affecting
each other.)
When we were designing our Data Modification Kit, we considered two
extreme examples: small storage devices (e.g., your 256MB USB drive) and
very large storage devices (e.g., your 16TB SQL database). Trying to
design to handle both extremes on this spectrum is a challenging one and
one you might not even understand until after you’ve developed your
version 1.0 product.
Another example: the NTFS team has some interesting challenges in the
coming years. The NTFS implementation has a documented limit (this may
change in Vista) of 16TB in a single volume (the on-disk format
limitation is higher than this.) Certainly this limitation hasn’t been
much of a big issue in the past - but we now see commercially available
1TB hard disk drives, which means you could almost trivially configure a
multi-terabyte disk array on a desktop or even a modest sized server
configuration. I’ve talked to multiple people who have (or support)
machines with 1TB of physical memory (64-bit processor boxes).
Developing even a simple file system is a daunting task. Designing a
file system that works in a broad range of conditions is something that
requires a tremendous amount of research (and it benefits from
experience as well). If your boss wants an NTFS clone, I’d be asking
"why? NTFS already exists. What’s the compelling case for reinventing
it? What makes you believe that you’ll be able to do better than a team
of people who have dedicated the past 17 years of their lives to
building, extending, maintaining and bug-fixing the existing NTFS
implementation? To be compelling it needs to provide something
above-and-beyond what’s already there.
File systems that make sense to me (compelling reason):
-
Clustered (distributed) file systems. Shared storage is complex but
CAN be done (talk to the Polyserve folks, for instance.) Many have
tried but few have succeeded here. Tough problem (which makes it fun). -
Legacy file system. Want to read HFS disks on your Windows box?
While it’s probably easier to do a one-time conversion, there are people
who periodically ask for this (we’ve done it in the past.) UDFS is
marginal at this point (most UDF writers do it at packet level and then
use the native reader.) Vista includes a UDF 2.5 read/write. I don’t
see supporting UDF 2.6 compelling reason to do a new implementation.
But building NTFS or FAT all over again doesn’t seem compelling. -
Network file system. SMB is an atrocious network procotol. MS has
done a remarkable job of extending it and making it work but it suffers
from some serious limitations (server failure recovery models in SMB are
primitive at best.) But I can see compelling reasons for people to look
at building NFS V4 implementations for Windows at this point.
Filters may be compelling as well, but generally because they provide
some sort of functionality on top of the existing file systems. Filters
often prove to be MORE difficult to develop successfully than file
systems. I generally explain that this is because it has an
undocumented and very complex upper edge interface (the IRPs arriving in
the filter) and an undocumented and very complex lower edge interface
(the IRPs sent to the next filter or underlying file system). Filter
Manger simplifies some aspects of this (much like our own filter driver
kit simplifies some aspects of this in the legacy environment,) but
doesn’t take away the need to understand the semantics of operations.
It’s sort of a “fixed decrease in overhead” model. So that simple
filters are much simpler, but complex filters are only a little bit
simpler. Anti-virus products, quota checkers, real-time backup
products, security products (policy managers, encryption, system
protection, ACLs, etc,) all make sense and people continue to write them
(and are likely to continue writing them in the foreseeable future.)
You mentioned W2K so I’ll ignore the interesting challenges that will be
arriving with Vista (e.g., transactions).
My goal isn’t to discourage you - I’ve been doing this for many years (I
started working on my first physical media file system the same year the
NTFS team started developing NTFS) and I still find the field
fascinating and challenging (although, I admit I like to pick the
interesting projects - when I started the design on our DMK two years
ago I did so because I’d figured out how to solve the hard problem in
data modification - transparently supporting compression. See, if you
update data in-place it might not fit there any longer, but users can’t
be aware that this really is the case.) The interesting challenges like
this are what make this business so much fun! But you don’t want to go
into this without understanding that this is a very complex area in
which to work.
Best of luck on your project, however you decide to proceed on it.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of like
Sent: Thursday, March 02, 2006 12:29 AM
To: ntfsd redirect
Subject: Re: Re:[ntfsd] Any example about ntfs file system?
Thanks for your suggestion.
I must accomplish the task because it is my job that my boss give me.
I would like the winning lottery ticket that was sold in Ohio too. I
don’t
expect to get it and you shouldn’t either. Developing a FSD is a major
task
that very few can accomplish. OSR sells a development kit that have
samples.“like” wrote in message news:xxxxx@ntfsd…
>>
>> I need to develop a file system based ntfs on win2k.But I can’t find
any
>> example. Where Can I find any suggestions?
>> Thanks.
>>
>>
>>
>>
>
>
>
>—
>Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
>You are currently subscribed to ntfsd as: likeosr@126.com
>To unsubscribe send a blank email to xxxxx@lists.osr.com
>
= = = = = = = = = = = = = = = = = = = =
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Thanks Tony. Also even though FastFat has been around since Windows 2000 in
its current form. Earlier versions didn’t have to support as much in the
way of removable media and removable drives. Try writing good code for
devices that have removable media and are physically removable too. There
is at least one hotfix I know of that corrects a specific problem where a
device is surprise removed and a bug check happens when a object is used.
Some of these problems are showing up now because so many systems are
becoming so much faster that some events can happen on other CPUs while
another one is plodding along tearing down stuff.
If you try a file system make sure you have something like the Tyan
motherboard that supports 8 Opteron dual core processors and 128GB of
memory. Some major server vendors make similar systems. Run your FSD for a
month running all the mass storage and file system HCT tests. You want the
system to respond like an old IBM XT with a single floppy drive and no hard
disk while these tests are running. Create on several very large SCSI 320
drives your partition and run the tests on a single volume and then on at
least 8 volumes.
Hardware costs: Tyan is about $1400 bare without the daughter card. Add
the daughter card, then 8 Opteron dual core processor at over $1000 each.
Good memory in 2GB ECC dimms that total 128GB will add another $12,800.
There goes somewhere around $40,000 for one test system before you add the
SCSI 320 drives.
If you have a simple file system that runs on very slow media, you can work
with slower systems, but always have at least 4 processors but don’t count
hyper-threading.
“Tony Mason” wrote in message news:xxxxx@ntfsd…
I think David’s point is that what you are asking about is an extremely
challenging task to accomplish. You can start with the IFS Kit (or the
Windows Vista WDK which subsumes the Windows Server 2003 SP 1 IFS Kit)
and examine the samples that it contains. Depending upon the
requirements of your project, you will find that a project to implement
a functional file system (similar to FAT, for example) would likely be
2-3 years of time for a single developer, particularly one who has to
learn about Windows file systems. Someone who has experience developing
Windows file systems can likely accomplish the task in around 1 year.
As you increase the requirements and complexity of the project, you will
quickly see that time estimate increase (one project on which I worked
in the past included 125 engineering years of effort for the Version 1.0
product.)
This may not be in line with your boss’s expectations. If that is the
case, I would suggest that you educate your boss, find a different boss,
or prepare to have him blame you when you are not able to meet his
(unrealistic) expectations. I know it is often difficult to deliver bad
news to a boss, but it is far better to educate him now than it will be
to have him think that you are incompetent when you must struggle to
understand the rather complex file systems model inside Windows. If,
for instance, your boss believes this is comparable to developing a
Linux file system, your boss would be incorrect - it is considerably
more complex, in no small part because of the file systems model that
has been adopted by the Windows team.
Assuming that your boss wants you to develop a file system “from
scratch” and does not consider the cost of your time in developing it to
have any value, my suggestion would be to obtain a copy of the Windows
Server 2003 SP1 IFS Kit (from Microsoft, US $109 plus shipping and
handling) and a copy of the Windows NT File Systems Development book
(which OSR offers in a reprint version available on the OSR Online
store, or you can purchase in the used book market.) If your boss is
unwilling to expend the 2-3 year effort for you to obtain on-the-job
training, I would suggest that you consider taking the file systems
class (I’m teaching again in April) next month. Generally, I’ve been
told by people who have taken the class that it has helped their
development process immensely (I have several people who have taken it
more than once - there is a 50% discount to take the class a second time
- and who have told me that each time they have learned more from the
class.)
If your boss is more concerned about market pressures (e.g. “time to
market” and “total cost of ownership”) then he might be better off
considering working with any one of the consultants that work in this
space. While OSR does work in this space, we typically only engage in
complex technical projects and from your brief description it does not
sound like your project is technically complex. In addition there are
several other people who can also work with you, several of whom
participate in this forum.
On the positive side for you, the skills to develop a file system are
valuable. There are few experienced developers, and only a handful in
the world who have been involved in multiple file system development
projects. My experience is that companies often MUST train their own
staff because they cannot find anyone with these specialized skills.
File systems is a challenging and complex area in which to work. Unlike
application programming (where there is often a concept of “close
enough”) it is intolerant of errors. I usually compare it to the
“payroll department” in your company - people do not tolerate errors in
the payroll process (“oh, gee we forgot to pay you this month. No
worries, we’ll catch up next month” is similar to “oh, gee we lost your
data. No worries, we’ll keep the next file you give to us”.) And
almost all of the really interesting parts of file systems development
relate to handling failures - failures in the network, the storage
system, this computer, other computers. Byzantine failures are
particularly “fun” - at FAST '05 Jim Gray (Microsoft Research) talked
about how the Exchange team was noticing a class of disk errors that
neither the NTFS nor the SQL team observed. It turns out that Exchange
implements an end-to-end checksum on the data that they write to disk,
while NTFS and SQL do not. As a result, Exchange was noticing errors
where the data returned from the disk was not what was written to the
disk. (This is increasingly common, as disk drive densities become
higher and higher there are issues related to adjacent bits affecting
each other.)
When we were designing our Data Modification Kit, we considered two
extreme examples: small storage devices (e.g., your 256MB USB drive) and
very large storage devices (e.g., your 16TB SQL database). Trying to
design to handle both extremes on this spectrum is a challenging one and
one you might not even understand until after you’ve developed your
version 1.0 product.
Another example: the NTFS team has some interesting challenges in the
coming years. The NTFS implementation has a documented limit (this may
change in Vista) of 16TB in a single volume (the on-disk format
limitation is higher than this.) Certainly this limitation hasn’t been
much of a big issue in the past - but we now see commercially available
1TB hard disk drives, which means you could almost trivially configure a
multi-terabyte disk array on a desktop or even a modest sized server
configuration. I’ve talked to multiple people who have (or support)
machines with 1TB of physical memory (64-bit processor boxes).
Developing even a simple file system is a daunting task. Designing a
file system that works in a broad range of conditions is something that
requires a tremendous amount of research (and it benefits from
experience as well). If your boss wants an NTFS clone, I’d be asking
"why? NTFS already exists. What’s the compelling case for reinventing
it? What makes you believe that you’ll be able to do better than a team
of people who have dedicated the past 17 years of their lives to
building, extending, maintaining and bug-fixing the existing NTFS
implementation? To be compelling it needs to provide something
above-and-beyond what’s already there.
File systems that make sense to me (compelling reason):
- Clustered (distributed) file systems. Shared storage is complex but
CAN be done (talk to the Polyserve folks, for instance.) Many have
tried but few have succeeded here. Tough problem (which makes it fun).
- Legacy file system. Want to read HFS disks on your Windows box?
While it’s probably easier to do a one-time conversion, there are people
who periodically ask for this (we’ve done it in the past.) UDFS is
marginal at this point (most UDF writers do it at packet level and then
use the native reader.) Vista includes a UDF 2.5 read/write. I don’t
see supporting UDF 2.6 compelling reason to do a new implementation.
But building NTFS or FAT all over again doesn’t seem compelling.
- Network file system. SMB is an atrocious network procotol. MS has
done a remarkable job of extending it and making it work but it suffers
from some serious limitations (server failure recovery models in SMB are
primitive at best.) But I can see compelling reasons for people to look
at building NFS V4 implementations for Windows at this point.
Filters may be compelling as well, but generally because they provide
some sort of functionality on top of the existing file systems. Filters
often prove to be MORE difficult to develop successfully than file
systems. I generally explain that this is because it has an
undocumented and very complex upper edge interface (the IRPs arriving in
the filter) and an undocumented and very complex lower edge interface
(the IRPs sent to the next filter or underlying file system). Filter
Manger simplifies some aspects of this (much like our own filter driver
kit simplifies some aspects of this in the legacy environment,) but
doesn’t take away the need to understand the semantics of operations.
It’s sort of a “fixed decrease in overhead” model. So that simple
filters are much simpler, but complex filters are only a little bit
simpler. Anti-virus products, quota checkers, real-time backup
products, security products (policy managers, encryption, system
protection, ACLs, etc,) all make sense and people continue to write them
(and are likely to continue writing them in the foreseeable future.)
You mentioned W2K so I’ll ignore the interesting challenges that will be
arriving with Vista (e.g., transactions).
My goal isn’t to discourage you - I’ve been doing this for many years (I
started working on my first physical media file system the same year the
NTFS team started developing NTFS) and I still find the field
fascinating and challenging (although, I admit I like to pick the
interesting projects - when I started the design on our DMK two years
ago I did so because I’d figured out how to solve the hard problem in
data modification - transparently supporting compression. See, if you
update data in-place it might not fit there any longer, but users can’t
be aware that this really is the case.) The interesting challenges like
this are what make this business so much fun! But you don’t want to go
into this without understanding that this is a very complex area in
which to work.
Best of luck on your project, however you decide to proceed on it.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of like
Sent: Thursday, March 02, 2006 12:29 AM
To: ntfsd redirect
Subject: Re: Re:[ntfsd] Any example about ntfs file system?
Thanks for your suggestion.
I must accomplish the task because it is my job that my boss give me.
>
>I would like the winning lottery ticket that was sold in Ohio too. I
don’t
>expect to get it and you shouldn’t either. Developing a FSD is a major
task
>that very few can accomplish. OSR sells a development kit that have
>samples.
>
>“like” wrote in message news:xxxxx@ntfsd…
>>
>> I need to develop a file system based ntfs on win2k.But I can’t find
any
>> example. Where Can I find any suggestions?
>> Thanks.
>>
>>
>>
>>
>
>
>
>—
>Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
>You are currently subscribed to ntfsd as: likeosr@126.com
>To unsubscribe send a blank email to xxxxx@lists.osr.com
>
= = = = = = = = = = = = = = = = = = = =
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Hi ‘likeosr’,
— like wrote:
> Thanks for your suggestion.
> I must accomplish the task because it is my job
> that my boss give me.
This is doable. I also agree with Tony and David that
it is not simple. You should expect at the very least
a few months of ramping up and many more for the
development.
Why don’t you start with Rajeev Nagar’s book. It is
dated but will get you off to a start. You should also
get the IFS kit and look at the fastfat and other
sample FSDs.
All the best!
Regards,
Manoj
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Manoj Paul Joseph wrote:
Hi ‘likeosr’,
— like wrote:
>
>> Thanks for your suggestion.
>> I must accomplish the task because it is my job
>> that my boss give me.
Perhaps you do not need to implement a file system (actually I think
about very few cases where you actually need a whole new file system)?
Perhaps what you need is a filter driver to modify certain behaviours of
an existing file system. Would be much easier.
If you need to add support for a “foreign” file system on NT , then, you
have to do it from scratch. However if your file system is very trivial
with very few features it can be done in less than one year, especially
if you already have some code to work on. The tricky part will be to
adapt it, for example, to the cache manager.
As already pointed out, the FAT sample in the IFS kit is a very good
starting point as well as the file system book.
Good luck.
–
Edouard