Hi, all!
Excuse me for off question, but I can’t find an answer
for two days. I have to concatenate two huge files. If
I use standard way (open two files; append source file
contents to destination file; delete source file), I have to
overcome low disk space problem (suppose I have 4Gb
HDD with only two 1.5Gb files).
Maybe there is a legal way to directly modify FATs and
directory?
S.Y.
Igor
> for two days. I have to concatenate two huge files. If
I use standard way (open two files; append source file
contents to destination file; delete source file), I have to
overcome low disk space problem (suppose I have 4Gb
HDD with only two 1.5Gb files).
Maybe there is a legal way to directly modify FATs and
directory?
No. This will not work on NTFS.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Huh? So you want to remove the beginning of the file before you get to the
end of it, and reuse those blocks for the beginning of the new file? Or just
link the end of one file to the beginning of another one?
The only way I can think of is that you split your 1.5GB file into much
smaller files (ignoring the fact that you’ll still have the problem that
your disk is getting full from doing this…), and then write a script that
concatenates files and remove the source file as part of the operation.
Ah, just thought of something. If you write a small application to take a
“chunk” out from the back of the file, and then set the end of file
“earlier”, using the set-end-of-file (_chsize(handle, size) in Visual C). So
then you’ll have a few smaller files that you can then apply the above “copy
and remove” operation on.
I’d suggest that you find a network drive with enough storage, but that’s
probably a bit too easy.
–
Mats
-----Original Message-----
From: Igor Slewsarev [mailto:xxxxx@yahoo.com]
Sent: Monday, June 07, 2004 2:02 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] How to concatenate files?Hi, all!
Excuse me for off question, but I can’t find an answer
for two days. I have to concatenate two huge files. If
I use standard way (open two files; append source file
contents to destination file; delete source file), I have to
overcome low disk space problem (suppose I have 4Gb
HDD with only two 1.5Gb files).
Maybe there is a legal way to directly modify FATs and
directory?S.Y.
Igor
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256You are currently subscribed to ntdev as: xxxxx@3dlabs.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Any hack that involves modification of FS data won’t do you much good
unless the first file has a length that is an exact multiple of the
minimum allocation unit size for the disk. Otherwise such a
concatenation will have “junk” between the end of the first file and the
beginning of the second file.
You might be able to do some trickery with NTFS sparse files, but
whether or not it would work depends on internal FS implementation
details. Convert both files to sparse files, copy pieces from file 2 to
file 1, zeroing out file 2 as you go, then convert file 1 back to a
non-sparse file. I don’t know if this is even possible, and it would
depend on the FS actively deallocating zeroed-out portions of the sparse
file.
Chuck
----- Original Message -----
From: “Igor Slewsarev”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Monday, June 07, 2004 8:01 PM
Subject: [ntdev] How to concatenate files?
> Hi, all!
>
> Excuse me for off question, but I can’t find an answer
> for two days. I have to concatenate two huge files. If
> I use standard way (open two files; append source file
> contents to destination file; delete source file), I have to
> overcome low disk space problem (suppose I have 4Gb
> HDD with only two 1.5Gb files).
> Maybe there is a legal way to directly modify FATs and
> directory?
>
> S.Y.
> Igor
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@cbatson.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
On Mon, 7 Jun 2004 17:20:42 +0400, you wrote:
> for two days. I have to concatenate two huge files. If
> I use standard way (open two files; append source file
> contents to destination file; delete source file), I have to
> overcome low disk space problem (suppose I have 4Gb
> HDD with only two 1.5Gb files).
> Maybe there is a legal way to directly modify FATs and
> directory?No. This will not work on NTFS.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
It won’t work on FAT, either, unless the first file is an exact multiple of
the cluster size.
Ralph
???
Is this a question from one or another exam at school ?
When I was 18, I would have solved it this way :
If you are working on a FAT system, you could put together both cluster chains … but that requires you to patch the FAT’s and
the directory entries.
----- Original Message -----
From: “Igor Slewsarev”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Monday, June 07, 2004 3:01 PM
Subject: [ntdev] How to concatenate files?
> Hi, all!
>
> Excuse me for off question, but I can’t find an answer
> for two days. I have to concatenate two huge files. If
> I use standard way (open two files; append source file
> contents to destination file; delete source file), I have to
> overcome low disk space problem (suppose I have 4Gb
> HDD with only two 1.5Gb files).
> Maybe there is a legal way to directly modify FATs and
> directory?
>
> S.Y.
> Igor
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@compaqnet.be
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
If it is a general purpose solution you are looking for, you might want to
try some variation of file Merging. Assuming you have .5GB or some amount
(multiple of 512 bytes or a cluster size), we take a first block of the
source, and save at the end of the destination, delete the src block,
commit, and go on, after opening the file with file mapping …
Just a thought !
-pro
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Igor Slewsarev
Sent: Monday, June 07, 2004 6:02 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] How to concatenate files?
Hi, all!
Excuse me for off question, but I can’t find an answer
for two days. I have to concatenate two huge files. If
I use standard way (open two files; append source file
contents to destination file; delete source file), I have to
overcome low disk space problem (suppose I have 4Gb
HDD with only two 1.5Gb files).
Maybe there is a legal way to directly modify FATs and
directory?
S.Y.
Igor
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
You are currently subscribed to ntdev as: xxxxx@garlic.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
It boils down to the question, whether there is a good solution to
shorten a file from the beginning. Can you somehow push the
start_of_file into direction of the end_of_file?
For FAT-filesystems you may modify the dir_entry 'starting cluster' +
cluster-chain in the FAT.
1.) open first file, lseek to end.
2.) open second file, read N*clustersize to ram. close file.
3.) push starting_cluster of second file a distance of
N clusters ( follow the chain).
4.) write first file from ram.
5.) if size of second file drops below N*clustersize.
do a normal concat else goto 2.).
"Always chack your work."
---- snip ----
“Chuck Batson” wrote in message news:xxxxx@ntdev…
> Any hack that involves modification of FS data won’t do you much good
> unless the first file has a length that is an exact multiple of the
> minimum allocation unit size for the disk. Otherwise such a
> concatenation will have “junk” between the end of the first file and the
> beginning of the second file.
Cluster size alignment is suitable for me.
> You might be able to do some trickery with NTFS sparse files, but
> whether or not it would work depends on internal FS implementation
> details. Convert both files to sparse files, copy pieces from file 2 to
> file 1, zeroing out file 2 as you go, then convert file 1 back to a
> non-sparse file. I don’t know if this is even possible, and it would
> depend on the FS actively deallocating zeroed-out portions of the sparse
> file.
That’s what I’m doing if I don’t have enough disk space.
Great thanks
S.Y.
Igor
Too difficult, isn’t it? Suppose you have 1,5Gb file on WD IDE pc card.
You’ll be starring at copy progress dialog for several minutes (hours?).
From the over hand for FAT fs all we have to do - rewrite two bytes
in 1st and 2nd FAT and remove directory entry. More easier, isn’t it?
I simply hope that there is a legal way to do that.
wrote in message news:xxxxx@ntdev…
> Huh? So you want to remove the beginning of the file before you get to the
> end of it, and reuse those blocks for the beginning of the new file? Or
just
> link the end of one file to the beginning of another one?
Yes, I just want to link one file to the end of the another, and any
alignment
is siutable (I can convert gap to junk chunk(s)).
> The only way I can think of is that you split your 1.5GB file into much
> smaller files (ignoring the fact that you’ll still have the problem that
> your disk is getting full from doing this…), and then write a script
that
> concatenates files and remove the source file as part of the operation.
To my great regret, it’s not I, who wrotes 1.5Gb files; (I hope your
understand
my english).
> I’d suggest that you find a network drive with enough storage, but that’s
> probably a bit too easy.
Some parts of our software works on stand-alone (not connected to network)
industrial PCs.
Great thanks
Igor.
> Too difficult, isn’t it? Suppose you have 1,5Gb file on WD
IDE pc card.
You’ll be starring at copy progress dialog for several
minutes (hours?).
From the over hand for FAT fs all we have to do - rewrite two bytes
in 1st and 2nd FAT and remove directory entry. More easier, isn’t it?
I simply hope that there is a legal way to do that.
The legal way to do it is to read some data from one file, and write it at
the end of the other file, in one way or another.
The reason that there isn’t an generic implementation of “chain the end of
this file to the end of some other file” is that this is in fact a fairly
rare operation, that can easily be achieved with a simple matter of copying
one of the files to the end of the other with a simple copy command. Your
problem is that the hard disk isn’t big enough. I don’t see why this should
be made into a big issue…
We’ve suggested several solutions, which are all legal from user-mode,
without going through trickery of doing nasty hacks in the directory entry.
But I’m sure if you boot your machine into DOS-mode and run Norton disk
editor, you can patch up the files together, at least in the simple form
that you just change the fat entry of one file to point to the beginning of
the next file. Just don’t chkdsk the disk until you’ve stored the file
somewhere safe…
–
Mats
Only if the size of the first file is strictly the multiple of the cluster
size.
Due to this limitation, no OS I know provides any way of file concatenation
by metadata tricks.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: “Igor Slewsarev”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Monday, June 07, 2004 6:55 PM
Subject: Re:[ntdev] How to concatenate files?
> Too difficult, isn’t it? Suppose you have 1,5Gb file on WD IDE pc card.
> You’ll be starring at copy progress dialog for several minutes (hours?).
> From the over hand for FAT fs all we have to do - rewrite two bytes
> in 1st and 2nd FAT and remove directory entry. More easier, isn’t it?
> I simply hope that there is a legal way to do that.
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
Ideally it should be at the byte level, so for example if both of the files are not of some multiple of cluster/sector size ( valid data size, might not be the file size shows in dir ) STILL IT SHOULD BE AND MUST BE POSSIBLE to have the stream semantic of C file i/o and there must be a way to chop off random number of bytes sequentiall ( an example is an editor ), if that is possible, than that chopped off data is easy enough to append to the target file. To have a clean soution, w/o the underlying file system, the start file pointer should be manipulated to effect a delete, and probably fstream is a good candidate to look at !!!
-pro
> Yes, I just want to link one file to the end of the another, and any
alignment
is siutable (I can convert gap to junk chunk(s)).
This implies you have some knowledge, and perhaps control, over the
format of the files in question. If this is true, why not add a chunk
to the end of the first file that says, “continue with the second file
transparently as if it was one big file”?
Chuck
> I can convert gap to junk chunk(s)
Can you do so in every possible case?
E.g. the gap being only 1, 2, 3… Bytes?
Or can such a case never happen?
Just make sure you have some fallback-
mechanism in case it could happen…
bye,
Paul Groke
> Can you do so in every possible case?
E.g. the gap being only 1, 2, 3… Bytes?
I hope in this case I could add one more cluster.
Thanks for response
Igor