Writing to the middle of a file

I would like to be able to write to the middle of a file without
overwritting the contents of the middle of the file. Essentially expanding
the file from the middle. I originally thought I could do this with the
“defrag” api’s (DeviceIoControl / FSCTL_MOVE_FILE), but it doesn’t look
possible. I know that I can extend the file and manually move the data and
then write to the middle, but this seems expensive especially if it’s a
large file. Seems like it would be much more efficient to modify the
cluster/MFT pointers. Is there a way I can do this from user mode or will I
need to write a file system filter driver? Or is there some other suggested
way of doing this?

Thanks…
-Gerald

Think of using NTFS sparse files… the ‘zero’d out’ areas are not
actually allocated on disk until you decide to use them.

Gerald Schwab wrote:

I would like to be able to write to the middle of a file without
overwritting the contents of the middle of the file. Essentially expanding
the file from the middle. I originally thought I could do this with the
“defrag” api’s (DeviceIoControl / FSCTL_MOVE_FILE), but it doesn’t look
possible. I know that I can extend the file and manually move the data and
then write to the middle, but this seems expensive especially if it’s a
large file. Seems like it would be much more efficient to modify the
cluster/MFT pointers. Is there a way I can do this from user mode or will I
need to write a file system filter driver? Or is there some other suggested
way of doing this?

Thanks…
-Gerald


Nick Ryan (MVP for DDK)

Thanks Nick,

I thought about using sparse files, but I would like the file to behave as
much like a normal file as possible. Many file utilities(even windows
explorer if I’m not mistaken) are un-aware that they are dealing with sparse
files. Is there a way to manipulate the MFT cluster pointers? I realize
this would lead to quick fragmentation of the file, but it could be defagged
using a low proirity thread.

Thanks…
-Gerald

“Nick Ryan” wrote in message news:xxxxx@ntfsd…
>
> Think of using NTFS sparse files… the ‘zero’d out’ areas are not
> actually allocated on disk until you decide to use them.
>
> Gerald Schwab wrote:
>
> > I would like to be able to write to the middle of a file without
> > overwritting the contents of the middle of the file. Essentially
expanding
> > the file from the middle. I originally thought I could do this with the
> > “defrag” api’s (DeviceIoControl / FSCTL_MOVE_FILE), but it doesn’t look
> > possible. I know that I can extend the file and manually move the data
and
> > then write to the middle, but this seems expensive especially if it’s a
> > large file. Seems like it would be much more efficient to modify the
> > cluster/MFT pointers. Is there a way I can do this from user mode or
will I
> > need to write a file system filter driver? Or is there some other
suggested
> > way of doing this?
> >
> > Thanks…
> > -Gerald
> >
> >
> >
> >
>
> –
> Nick Ryan (MVP for DDK)
>
>
>

----- Original Message -----
From: “Gerald Schwab”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Monday, September 15, 2003 11:56 PM
Subject: [ntfsd] Re: Writing to the middle of a file

> Thanks Nick,
>
> I thought about using sparse files, but I would like the file to behave as
> much like a normal file as possible. Many file utilities(even windows
> explorer if I’m not mistaken) are un-aware that they are dealing with
sparse
> files. Is there a way to manipulate the MFT cluster pointers? I realize
> this would lead to quick fragmentation of the file, but it could be
defagged
> using a low proirity thread.

At the risk of pointing out the obvious, the file model used in NT (and
'most anywhere else: I seem to remember an experimental physical database
layer at UWisc that supported such insertions, but aside from that…)
doesn’t lend itself to such actions. Specific issues include the change to
the file’s EOF position, its ‘valid data length’, possible interactions with
quota mechanisms (I’d expect normal defrag activity to be able to ignore
them), and the file offsets of any current accessors (at or beyond the point
of insert). While the file and access to it may be sufficiently under your
control that you can ignore some of these areas, they may help explain why
the defrag mechanisms don’t seem to be set up to allow this.

- bill

Design your own database-like format for the file to add data in the middle.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Gerald Schwab”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Tuesday, September 16, 2003 7:56 AM
Subject: [ntfsd] Re: Writing to the middle of a file

> Thanks Nick,
>
> I thought about using sparse files, but I would like the file to behave as
> much like a normal file as possible. Many file utilities(even windows
> explorer if I’m not mistaken) are un-aware that they are dealing with sparse
> files. Is there a way to manipulate the MFT cluster pointers? I realize
> this would lead to quick fragmentation of the file, but it could be defagged
> using a low proirity thread.
>
> Thanks…
> -Gerald
>
> “Nick Ryan” wrote in message news:xxxxx@ntfsd…
> >
> > Think of using NTFS sparse files… the ‘zero’d out’ areas are not
> > actually allocated on disk until you decide to use them.
> >
> > Gerald Schwab wrote:
> >
> > > I would like to be able to write to the middle of a file without
> > > overwritting the contents of the middle of the file. Essentially
> expanding
> > > the file from the middle. I originally thought I could do this with the
> > > “defrag” api’s (DeviceIoControl / FSCTL_MOVE_FILE), but it doesn’t look
> > > possible. I know that I can extend the file and manually move the data
> and
> > > then write to the middle, but this seems expensive especially if it’s a
> > > large file. Seems like it would be much more efficient to modify the
> > > cluster/MFT pointers. Is there a way I can do this from user mode or
> will I
> > > need to write a file system filter driver? Or is there some other
> suggested
> > > way of doing this?
> > >
> > > Thanks…
> > > -Gerald
> > >
> > >
> > >
> > >
> >
> > –
> > Nick Ryan (MVP for DDK)
> >
> >
> >
>
>
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

> doesn’t lend itself to such actions. Specific issues include the change to

the file’s EOF position, its ‘valid data length’, possible interactions with
quota mechanisms (I’d expect normal defrag activity to be able to ignore
them), and the file offsets of any current accessors (at or beyond the point

Theoretically speaking, adding an API to split the file to 2 files (truncate
the tail to another file) and to merge the 2 files would be OK. A pity that
filesystems do not support it.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Thanks Maxim,

I thought about doing that also, but I’m trying to write to the middle of an
xml file. And as such, cannot use a database-like format.

Thanks…
-Gerald

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntfsd…
>
> Design your own database-like format for the file to add data in the
middle.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> ----- Original Message -----
> From: “Gerald Schwab”
> Newsgroups: ntfsd
> To: “Windows File Systems Devs Interest List”
> Sent: Tuesday, September 16, 2003 7:56 AM
> Subject: [ntfsd] Re: Writing to the middle of a file
>
>
> > Thanks Nick,
> >
> > I thought about using sparse files, but I would like the file to behave
as
> > much like a normal file as possible. Many file utilities(even windows
> > explorer if I’m not mistaken) are un-aware that they are dealing with
sparse
> > files. Is there a way to manipulate the MFT cluster pointers? I
realize
> > this would lead to quick fragmentation of the file, but it could be
defagged
> > using a low proirity thread.
> >
> > Thanks…
> > -Gerald
> >
> > “Nick Ryan” wrote in message news:xxxxx@ntfsd…
> > >
> > > Think of using NTFS sparse files… the ‘zero’d out’ areas are not
> > > actually allocated on disk until you decide to use them.
> > >
> > > Gerald Schwab wrote:
> > >
> > > > I would like to be able to write to the middle of a file without
> > > > overwritting the contents of the middle of the file. Essentially
> > expanding
> > > > the file from the middle. I originally thought I could do this with
the
> > > > “defrag” api’s (DeviceIoControl / FSCTL_MOVE_FILE), but it doesn’t
look
> > > > possible. I know that I can extend the file and manually move the
data
> > and
> > > > then write to the middle, but this seems expensive especially if
it’s a
> > > > large file. Seems like it would be much more efficient to modify
the
> > > > cluster/MFT pointers. Is there a way I can do this from user mode
or
> > will I
> > > > need to write a file system filter driver? Or is there some other
> > suggested
> > > > way of doing this?
> > > >
> > > > Thanks…
> > > > -Gerald
> > > >
> > > >
> > > >
> > > >
> > >
> > > –
> > > Nick Ryan (MVP for DDK)
> > >
> > >
> > >
> >
> >
> >
> > —
> > You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>

Thanks Bill,

Well unfortunately I think your point is the correct one. I doesn’t look
like writing to the middle of a file is possible without doing something
“special”(sparse file, propriety file format, etc…)

Thanks…
-Gerald

“Bill Todd” wrote in message news:xxxxx@ntfsd…
>
>
> ----- Original Message -----
> From: “Gerald Schwab”
> Newsgroups: ntfsd
> To: “Windows File Systems Devs Interest List”
> Sent: Monday, September 15, 2003 11:56 PM
> Subject: [ntfsd] Re: Writing to the middle of a file
>
>
> > Thanks Nick,
> >
> > I thought about using sparse files, but I would like the file to behave
as
> > much like a normal file as possible. Many file utilities(even windows
> > explorer if I’m not mistaken) are un-aware that they are dealing with
> sparse
> > files. Is there a way to manipulate the MFT cluster pointers? I
realize
> > this would lead to quick fragmentation of the file, but it could be
> defagged
> > using a low proirity thread.
>
> At the risk of pointing out the obvious, the file model used in NT (and
> 'most anywhere else: I seem to remember an experimental physical database
> layer at UWisc that supported such insertions, but aside from that…)
> doesn’t lend itself to such actions. Specific issues include the change
to
> the file’s EOF position, its ‘valid data length’, possible interactions
with
> quota mechanisms (I’d expect normal defrag activity to be able to ignore
> them), and the file offsets of any current accessors (at or beyond the
point
> of insert). While the file and access to it may be sufficiently under
your
> control that you can ignore some of these areas, they may help explain
why
> the defrag mechanisms don’t seem to be set up to allow this.
>
> - bill
>
>
>
>

Use some binary internal representation of the data, which will be
database-like.
Then generate XML from it on the fly when XML will be requested.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Gerald Schwab”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Tuesday, September 16, 2003 8:01 PM
Subject: [ntfsd] Re: Writing to the middle of a file

> Thanks Maxim,
>
> I thought about doing that also, but I’m trying to write to the middle of an
> xml file. And as such, cannot use a database-like format.
>
> Thanks…
> -Gerald
>
> “Maxim S. Shatskih” wrote in message
> news:xxxxx@ntfsd…
> >
> > Design your own database-like format for the file to add data in the
> middle.
> >
> > Maxim Shatskih, Windows DDK MVP
> > StorageCraft Corporation
> > xxxxx@storagecraft.com
> > http://www.storagecraft.com
> >
> >
> > ----- Original Message -----
> > From: “Gerald Schwab”
> > Newsgroups: ntfsd
> > To: “Windows File Systems Devs Interest List”
> > Sent: Tuesday, September 16, 2003 7:56 AM
> > Subject: [ntfsd] Re: Writing to the middle of a file
> >
> >
> > > Thanks Nick,
> > >
> > > I thought about using sparse files, but I would like the file to behave
> as
> > > much like a normal file as possible. Many file utilities(even windows
> > > explorer if I’m not mistaken) are un-aware that they are dealing with
> sparse
> > > files. Is there a way to manipulate the MFT cluster pointers? I
> realize
> > > this would lead to quick fragmentation of the file, but it could be
> defagged
> > > using a low proirity thread.
> > >
> > > Thanks…
> > > -Gerald
> > >
> > > “Nick Ryan” wrote in message news:xxxxx@ntfsd…
> > > >
> > > > Think of using NTFS sparse files… the ‘zero’d out’ areas are not
> > > > actually allocated on disk until you decide to use them.
> > > >
> > > > Gerald Schwab wrote:
> > > >
> > > > > I would like to be able to write to the middle of a file without
> > > > > overwritting the contents of the middle of the file. Essentially
> > > expanding
> > > > > the file from the middle. I originally thought I could do this with
> the
> > > > > “defrag” api’s (DeviceIoControl / FSCTL_MOVE_FILE), but it doesn’t
> look
> > > > > possible. I know that I can extend the file and manually move the
> data
> > > and
> > > > > then write to the middle, but this seems expensive especially if
> it’s a
> > > > > large file. Seems like it would be much more efficient to modify
> the
> > > > > cluster/MFT pointers. Is there a way I can do this from user mode
> or
> > > will I
> > > > > need to write a file system filter driver? Or is there some other
> > > suggested
> > > > > way of doing this?
> > > > >
> > > > > Thanks…
> > > > > -Gerald
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > –
> > > > Nick Ryan (MVP for DDK)
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > —
> > > You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> > > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
> >
>
>
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com