If my observations are correct then, valid data length and EOF use to have
same values till Windows 2000. There was no API ( Win32 or Nt)to set VDL and
MS provided Win32 API and NT API( IFS KIT) for setting EOF. Please correct
me, if I am wrong. One more thing, when one has set the EOF, FS fill up zero
and there was no way to avoid this.
With Window XP and Windows2003, these two values are used in a smarter ways.
in the same ways as Tony describes in his mail). Now, there are APIs to set
the VDL.
My understanding is that motivation comes from SAN. One of the usage is
following:
IBM and Veritas have implemented File Servers which are different than
typical file servers based NTFS & Windows technology. Instead of
transferring the data on LAN, client commits the data directly on physical
location bypassing the FS layer and FS server is in fact “just” the meta
data server.
Setting up VDL, without zeroing the regions, also has some utility for
backup & restore vendors. Think of server-less restores.
Regards
Pash
-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Monday, January 05, 2004 7:02 AM
To: Windows File Systems Devs Interest List
Subject: RE: Re:[ntfsd] Windows file system internals - Rajeev Nagar
Valid data length makes sense in a file system where you forcibly zero
out the regions of the file on disk. While you won’t notice much
difference for small or medium sized files, you will see a difference
for large files.
Also, you need to keep the historical perspective here (something that
sometimes gets lost). Compression did not show up in the original
version of NTFS (I believe it materialized in NT 3.51) and sparse file
support did not show up until Windows 2000.
Thus, Valid Data Length makes perfect sense for a file system that
supports large files (e.g., NTFS) in the absence of compression (which
eliminates the zero-filled regions anyway) or sparse file support (which
encodes them as part of the file layout description). Even now you can
observe this: create a large NTFS file system (I’d recommend > 1 TB to
really get the full effect here) and then create a large NTFS file
(again, 1 TB will give you a good sense of this) by setting the EOF to
be 1TB. So long as you don’t write data out to 1TB you won’t see a long
pause.
Now repeat this experiment but instead of moving EOF, write a single
byte at the 1TB location. Be prepared for a long pause (at a rate of
40MB/s you’d expect to take at least 7.5 hours to do this) as NTFS zeros
out all of the blocks on the disk up to that last single byte location.
In 1990, 1 TB was a huge amount of data storage (we were using “big” 1GB
disks at the time) and probably near the upper end of their design
center (I know that was the case for the journaling file system I helped
design and implement in that time period). But everyone knew disk
subsystems were slow, and demand-zeroing out large regions of files was
a performance bottleneck (we actually used a different technique for
ensuring zero-filled regions specifically to avoid the same hit because
otherwise it shows up on certain benchmarks.)
I very seldom find decisions in Windows that don’t make sense when
considered in their original context - and generally when they don’t
make sense to me it is because I do not understand the context in which
the design decision was made.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
Hope to see you at the next OSR file systems class in Boston, MA,
February 23, 2003!
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Monday, January 05, 2004 5:48 AM
To: ntfsd redirect
Subject: Re: Re:[ntfsd] Windows file system internals - Rajeev Nagar
If the ‘help’ that Microsoft would provide is the source code to the
file system ‘stuff’ and the storage stacks, it might be possible to do
something more than the current IFS Kit provides.
At least a clear documentation on how the FCB locks are used would be
good.
The suggestion in Rajeev’s book was a bit contradicting to FASTFAT
source IIRC.
Also - why use ValidDataLength at all? Why not mandate it to be always
equal to
EOF? Handling sparse tails? Sorry, but if the FSD supports sparse tails,
then
it supports sparse files in general, in the middle of the file too, and
this
support must be done by means of low-level IO routines and not the VDL.
So, why ever use VDL? Why not mandate it to be == EOF? Will it cause any
undesirable effects on Cache Manager?
How the Cache Manager handles the space which is >= VDL and < EOF?
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@legato.com
To unsubscribe send a blank email to xxxxx@lists.osr.com