fast file copy or read operation

NTFSD
1

Hi All,

I am thinking of an optimal solution for a Video application to speed up
these large file copy or video streaming.
something analogous to Direct-X in video. Where the application can call the
API that will move files or copy/read files bypassing the file system and
directly specifying our hardware the LBA start and the block size. The files
could be scattered at differrent physical locations on the disk, I am
wondering how to find out how many such fragments are? what is their
physical LBA locations on the disk?
I am trying here what you experts think the feasibility of this approach?
Will this speed up any file read/move/copy operations?

Thanks,
Shakeel

2

> Hi All,

I am thinking of an optimal solution for a Video application
to speed up
these large file copy or video streaming.
something analogous to Direct-X in video. Where the
application can call the
API that will move files or copy/read files bypassing the
file system and
directly specifying our hardware the LBA start and the block
size. The files
could be scattered at differrent physical locations on the disk, I am
wondering how to find out how many such fragments are? what is their
physical LBA locations on the disk?
I am trying here what you experts think the feasibility of
this approach?
Will this speed up any file read/move/copy operations?

Thanks,
Shakeel

I really doubt that bypassing NTFS will speed up big files read/write ops.
From my very limited testing NTFS gets 80-90% of the raw device write
bandwith.

Just open the files(s) with the flag FILE_FLAG_NO_BUFFERING.
Adding FILE_FLAG_SEQUENTIAL_SCAN and FILE_FLAG_OVERLAPPED
(using async overlapped IO) should help too.
Preallocating the space for the file can be useful.

Also you HAVE TO keep the file system defragmented.

Dmitriy

3

This is too general a question you dropped !
My suggestion would be to look at the windows media center, and windows
media sdk. The reason is that, there are too many personalities for media
streaming ( local, local area network(wired/wireless), wan ). Pay-per-view
but scheduled at certain prefixed time ( matinee, evening, night show some
thing like that …), production systems, random scheduling-on demand etc.,
etc., etc. And there are a wide number of areas to cover, including
conformiblity with the existing os(cache system,filesystem,disk managment)
…

you might want to look at the google for “Media server”, one of the best
site is from University of southern california, and you might get plenty of
information, actually mind bogling bleeding edge(s)…

-pro

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of shakeel
Sent: Monday, June 14, 2004 5:34 PM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] fast file copy or read operation

Hi All,

I am thinking of an optimal solution for a Video application to speed up
these large file copy or video streaming.
something analogous to Direct-X in video. Where the application can call the
API that will move files or copy/read files bypassing the file system and
directly specifying our hardware the LBA start and the block size. The files
could be scattered at differrent physical locations on the disk, I am
wondering how to find out how many such fragments are? what is their
physical LBA locations on the disk?
I am trying here what you experts think the feasibility of this approach?
Will this speed up any file read/move/copy operations?

Thanks,
Shakeel

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@garlic.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

4

----- Original Message -----
From: “shakeel”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Monday, June 14, 2004 8:34 PM
Subject: [ntfsd] fast file copy or read operation

> Hi All,
>
> I am thinking of an optimal solution for a Video application to speed up
> these large file copy or video streaming.
> something analogous to Direct-X in video. Where the application can call
the
> API that will move files or copy/read files bypassing the file system and
> directly specifying our hardware the LBA start and the block size. The
files
> could be scattered at differrent physical locations on the disk, I am
> wondering how to find out how many such fragments are? what is their
> physical LBA locations on the disk?
> I am trying here what you experts think the feasibility of this approach?
> Will this speed up any file read/move/copy operations?

In most cases probably not, at least not by much. But there is one where it
might: where you’re using an ATA disk with write-back caching disabled,
being able to issue large write requests (rather than the 64 KB requests
that the cache will tend to generate, unless it’s smart enough to amalgamate
them) could significant improve write bandwidth (missing a full disk rev
every 64 KB really bites).

- bill

5

----- Original Message -----
From: “shakeel”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Monday, June 14, 2004 8:34 PM
Subject: [ntfsd] fast file copy or read operation

> Hi All,
>
> I am thinking of an optimal solution for a Video application to speed up
> these large file copy or video streaming.
> something analogous to Direct-X in video. Where the application can call
the
> API that will move files or copy/read files bypassing the file system and
> directly specifying our hardware the LBA start and the block size. The
files
> could be scattered at differrent physical locations on the disk, I am
> wondering how to find out how many such fragments are? what is their
> physical LBA locations on the disk?
> I am trying here what you experts think the feasibility of this approach?
> Will this speed up any file read/move/copy operations?

Whoops - there’s also the possibility of adding significant amounts of
overhead when you try to roll your own file copy/move mechanism. IIRC the
NT ‘copy file’ function makes use of the ability to preallocate the file at
its final size without having to zero-fill it on creation, such that each
block of the new file is written only once rather than twice. And I doubt
that your situation justifies trying to bypass file creation and modify the
MFT and allocation bitmap yourself (not that this would likely gain you
anything over the copy file function, unless the no-write-back ATA situation
I described existed, in which case it would probably be easier to enable the
disk’s write-back cache yourself for the duration of the copy and use the
native NT interface functions to emulate copy-file’s behavior - in fact,
using them at least to create the output file at its final size - in all
environments - is one way to help ensure that it’s not fragmented, without
getting into LBA-level access at all).

- bill

6

> something analogous to Direct-X in video. Where the application can call the

API that will move files or copy/read files bypassing the file system and
directly specifying our hardware the LBA start and the block size. The files
could be scattered at differrent physical locations on the disk, I am
wondering how to find out how many such fragments are?

Just open the file in noncached mode. No need in bypassing the FSD, since the
only operation which the FSD will do in noncached mode is this very
fragmenting - necessary anyway.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

7

----- Original Message -----
From: “Maxim S. Shatskih”
To: “Windows File Systems Devs Interest List”
Sent: Tuesday, June 15, 2004 9:06 AM
Subject: Re: [ntfsd] fast file copy or read operation

> > something analogous to Direct-X in video. Where the application can call
the
> > API that will move files or copy/read files bypassing the file system
and
> > directly specifying our hardware the LBA start and the block size. The
files
> > could be scattered at differrent physical locations on the disk, I am
> > wondering how to find out how many such fragments are?
>
> Just open the file in noncached mode. No need in bypassing the FSD, since
the
> only operation which the FSD will do in noncached mode is this very
> fragmenting - necessary anyway.

Hmmm. If you just do normal read/writes uncached to perform a copy (or move
between multiple file systems, assuming that a move within the same file
system can simply be implemented as a rename), you’ll be losing any ability
to overlap operations (e.g., if the source and destination are on different
disks, or even if they’re on the same disk but you don’t want to use
multi-megabyte buffers). So that gets you into asynchronous
multi-buffering, which the cache would otherwise be doing for you
transparently.

My guess is still that CopyFile may be the best mechanism to use: if it
doesn’t use the cache directly, then it likely does asynchronous
multi-buffering on its own (I probably would, in that situation) - and IIRC
it does the file preallocation to full size optimization that I mentioned
earlier which you’d otherwise have to use native NT (non-Win32) calls to
achieve efficiently.

- bill

8

> API that will move files or copy/read files bypassing the file system and

directly specifying our hardware the LBA start and the block size. The
files
could be scattered at differrent physical locations on the disk, I am
wondering how to find out how many such fragments are?

This idea is bad from the ground. Even if you teoreticalle had
much time and money for this:

  • To do such a thing, you would have to write something
    as replacement of the file system. Because there are many
    file system (each file system has its unique sector structure),
    you would have to implement replacements of all common ones
    (NTFS, FASTFAT, CDFS, …)

  • NTFS is not documented, so doing a replacement of it
    would be an incredibly difficult thing

  • Any replacement of a file system will be slower or at the best as fast
    as the built-in ones. If we take the system cache into account,
    I doubt that you can create something faster than the built-in filesystems

  • I cannot imagine a way how you would solve the together accesses
    from your SW and e.g. operating system.

  • and many more.

L.

9

>The files

> could be scattered at different physical locations on the disk, I am
> wondering how to find out how many such fragments are?

You can use FSCTL to find this information. This is going to be a two step
process. First, you need to determine the extents for the file and then you
need to translate these extents to physical extents. For the latter part,
you can use volume IOCTL. Both these FSCTL and IOCTL are well documented.
FSCTL_GET_RETRIEVAL_POINTERS
IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS

> Something analogous to Direct-X in video. Where the application can call
the
> API that will move files or copy/read files bypassing the file system and
> Directly specifying our hardware the LBA start and the block size.

You may want to use pre-allocation of the destination file. Unfortunately,
API for doing so is only available on Windows XP onwards. I believe this can
be the optimum solution for the copy operation.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-178303-
xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Tuesday, June 15, 2004 6:07 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] fast file copy or read operation

> something analogous to Direct-X in video. Where the application can call
the
> API that will move files or copy/read files bypassing the file system and
> directly specifying our hardware the LBA start and the block size. The
files
> could be scattered at differrent physical locations on the disk, I am
> wondering how to find out how many such fragments are?

Just open the file in noncached mode. No need in bypassing the FSD, since
the
only operation which the FSD will do in noncached mode is this very
fragmenting - necessary anyway.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@legato.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

10

RE: [ntfsd] fast file copy or read operation>You may want to use pre-allocation
of the destination file. Unfortunately, API for doing so is only

available on Windows XP onwards.

ntdll!ZwCreateFile is available in all NTs.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

11

>-----Original Message-----

From: xxxxx@lists.osr.com [mailto:bounce-178374-
xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Wednesday, June 16, 2004 1:36 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] fast file copy or read operation

RE: [ntfsd] fast file copy or read operation>You may want to use pre-
allocation
of the destination file. Unfortunately, API for doing so is only
>available on Windows XP onwards.

ntdll!ZwCreateFile is available in all NTs.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

[Pashupati Kumar] Thanks! Does NTFS initialize the pre-allocated space with
Zero’s?

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@legato.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

12

> ntdll!ZwCreateFile is available in all NTs.

Will NTFS initialize the allocated space with Zero’s?

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-178374-
xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Wednesday, June 16, 2004 1:36 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] fast file copy or read operation

RE: [ntfsd] fast file copy or read operation>You may want to use pre-
allocation
of the destination file. Unfortunately, API for doing so is only
>available on Windows XP onwards.

ntdll!ZwCreateFile is available in all NTs.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@legato.com
To unsubscribe send a blank email to xxxxx@lists.osr.com