All,
I may have to write a file system for a embedded system. The
system currently uses NTFS for some 1.2terabyte RAID drives, but
ntfs.sys is hitting me with some awful delays on some writes. A
typical write is 768 sectors and takes about 18miliseconds. The
control software can tolerate up to 200ms for a write and we see
those occasionally but we also get some 1.2 second writes.
I am seeing the delays in the ntfs.sys driver and not from disk.sys.
Once my ntfs drive starts to get full it seems that ntfs.sys may try
to take space from the MFT. This is fine, but I see occasion that it
may take ntfs.sys as long as 1.2seconds to break up my write
request and send it to disk.sys. Disk.sys sends and returns each
of these in 18miliseconds.
Does this sound correct? 1.2 seconds for ntfs.sys to figure out
how to move the MFT on a 80% full freshly formatted drive? The
processor is an embedded pentium 200MHz.
Am I doomed to make my own file system? Are there alternatives?
Matt Edwards
If I’m understanding your situation correctly, you have ~960 GB on a 1.2 TB
RAID, using NTFS. Any FS that preallocates large structures on the disk as
part of the FS, which can be reclaimed for additional storage space, will
eventually run into this and have a performance hit. 1.2 seconds seems
really bad, though. Maybe you can work with MS to identify the evident
performance issue in the NTFS routine that gives back the MFT space for
storage?
You haven’t mentioned the nature of the data. Is it in millions of small
files, or one (or a few) gigantic files. If you are working with a small
number of very large files, the right answer is to use the disk as RAW and
manage your data placement yourself. This is not eaxactly a full-blown FS,
as you don’t have to implement the whole FS driver, just the parts you need
directly in your app.
Or the quick and dirty answer is to decrease your utilization to ~50%,
either by relocating data, or by increasing your RAID to 2X the maximum
estimated usage.
Phil
Matt Edwards @lists.osr.com on 03/14/2002 10:55:17
AM
Please respond to “File Systems Developers”
Sent by: xxxxx@lists.osr.com
To: “File Systems Developers”
cc:
Subject: [ntfsd] ntfs question
All,
I may have to write a file system for a embedded system. The
system currently uses NTFS for some 1.2terabyte RAID drives, but
ntfs.sys is hitting me with some awful delays on some writes. A
typical write is 768 sectors and takes about 18miliseconds. The
control software can tolerate up to 200ms for a write and we see
those occasionally but we also get some 1.2 second writes.
I am seeing the delays in the ntfs.sys driver and not from disk.sys.
Once my ntfs drive starts to get full it seems that ntfs.sys may try
to take space from the MFT. This is fine, but I see occasion that it
may take ntfs.sys as long as 1.2seconds to break up my write
request and send it to disk.sys. Disk.sys sends and returns each
of these in 18miliseconds.
Does this sound correct? 1.2 seconds for ntfs.sys to figure out
how to move the MFT on a 80% full freshly formatted drive? The
processor is an embedded pentium 200MHz.
Am I doomed to make my own file system? Are there alternatives?
Matt Edwards
—
You are currently subscribed to ntfsd as: xxxxx@Seagate.com
To unsubscribe send a blank email to %%email.unsub%%
Phil,
Thanks for the reply. The files are large and I think RAW access
would be fine. Do you know of any sample drivers that act as a file
system and translate to RAW access?
Through the complexity of ‘big time important software architects’
I’m limited to using ZwCreateFile and ZwWriteFile. Oddly enough,
the file system is my choice.
Matt
Date sent: Thu, 14 Mar 2002 11:29:28 -0700
From: xxxxx@seagate.com
Subject: [ntfsd] Re: ntfs question
To: File Systems Developers
Send reply to: File Systems Developers
>
> If I’m understanding your situation correctly, you have ~960 GB on a 1.2 TB
> RAID, using NTFS. Any FS that preallocates large structures on the disk as
> part of the FS, which can be reclaimed for additional storage space, will
> eventually run into this and have a performance hit. 1.2 seconds seems
> really bad, though. Maybe you can work with MS to identify the evident
> performance issue in the NTFS routine that gives back the MFT space for
> storage?
>
> You haven’t mentioned the nature of the data. Is it in millions of small
> files, or one (or a few) gigantic files. If you are working with a small
> number of very large files, the right answer is to use the disk as RAW and
> manage your data placement yourself. This is not eaxactly a full-blown FS,
> as you don’t have to implement the whole FS driver, just the parts you need
> directly in your app.
>
> Or the quick and dirty answer is to decrease your utilization to ~50%,
> either by relocating data, or by increasing your RAID to 2X the maximum
> estimated usage.
>
> Phil
>
>
>
>
>
> Matt Edwards @lists.osr.com on 03/14/2002 10:55:17
> AM
>
> Please respond to “File Systems Developers”
>
> Sent by: xxxxx@lists.osr.com
>
>
> To: “File Systems Developers”
> cc:
>
> Subject: [ntfsd] ntfs question
>
>
> All,
>
> I may have to write a file system for a embedded system. The
> system currently uses NTFS for some 1.2terabyte RAID drives, but
> ntfs.sys is hitting me with some awful delays on some writes. A
> typical write is 768 sectors and takes about 18miliseconds. The
> control software can tolerate up to 200ms for a write and we see
> those occasionally but we also get some 1.2 second writes.
>
> I am seeing the delays in the ntfs.sys driver and not from disk.sys.
> Once my ntfs drive starts to get full it seems that ntfs.sys may try
> to take space from the MFT. This is fine, but I see occasion that it
> may take ntfs.sys as long as 1.2seconds to break up my write
> request and send it to disk.sys. Disk.sys sends and returns each
> of these in 18miliseconds.
>
> Does this sound correct? 1.2 seconds for ntfs.sys to figure out
> how to move the MFT on a 80% full freshly formatted drive? The
> processor is an embedded pentium 200MHz.
>
> Am I doomed to make my own file system? Are there alternatives?
>
> Matt Edwards
>
>
> —
> You are currently subscribed to ntfsd as: xxxxx@Seagate.com
> To unsubscribe send a blank email to %%email.unsub%%
>
>
>
>
> —
> You are currently subscribed to ntfsd as: xxxxx@thebayoffice.com
> To unsubscribe send a blank email to %%email.unsub%%
Try reformatting with a bigger cluster size and doing unbuffered i/o
-----Original Message-----
From: Matt Edwards [mailto:xxxxx@thebayoffice.com]
Sent: Thursday, March 14, 2002 9:55 AM
To: File Systems Developers
Subject: [ntfsd] ntfs question
All,
I may have to write a file system for a embedded system. The
system currently uses NTFS for some 1.2terabyte RAID drives, but
ntfs.sys is hitting me with some awful delays on some writes. A
typical write is 768 sectors and takes about 18miliseconds. The
control software can tolerate up to 200ms for a write and we see
those occasionally but we also get some 1.2 second writes.
I am seeing the delays in the ntfs.sys driver and not from disk.sys.
Once my ntfs drive starts to get full it seems that ntfs.sys may try
to take space from the MFT. This is fine, but I see occasion that it
may take ntfs.sys as long as 1.2seconds to break up my write
request and send it to disk.sys. Disk.sys sends and returns each
of these in 18miliseconds.
Does this sound correct? 1.2 seconds for ntfs.sys to figure out
how to move the MFT on a 80% full freshly formatted drive? The
processor is an embedded pentium 200MHz.
Am I doomed to make my own file system? Are there alternatives?
Matt Edwards
You are currently subscribed to ntfsd as: xxxxx@congruent.com
To unsubscribe send a blank email to %%email.unsub%%
Actually, I meant, “Don’t format the partition, and change your user-mode
App to CreateFile on the partition.”
Does that make it more clear that RAW disk access does not require a kernel
mode driver? You can certainly do raw disk access from kernel mode, but
I’m not sure why you want to do so. Either I’m not understanding your
problem description. or someone is trying to make your solution a lot more
difficult than it needs to be.
Hope this is more helpful than confusing,
Phil
Matt Edwards @lists.osr.com on 03/14/2002 12:13:06
PM
Please respond to “File Systems Developers”
Sent by: xxxxx@lists.osr.com
To: “File Systems Developers”
cc:
Subject: [ntfsd] Re: ntfs question
Phil,
Thanks for the reply. The files are large and I think RAW access
would be fine. Do you know of any sample drivers that act as a file
system and translate to RAW access?
Through the complexity of ‘big time important software architects’
I’m limited to using ZwCreateFile and ZwWriteFile. Oddly enough,
the file system is my choice.