sparse file problem...

NTFSD
1

Hi all,

I’m currently having a strange problem with a sparse file.

I don’t know if this is normal or not, but here is my problem:

1- I create an empty sparse file of 200KB.
2- I write 200 bytes to the second block of the sparse file (64KB + some offset)

The problem I then see is that the system overwrites the first block of my sparse file, and fills it with 0…

If I write to the third block, I don’t see this problem, but every time I do this on the second block, the first block is 0 filled…

Also, I only see this on one of my machines, which is 2003 server R2 sp2…

Although I’ve only tested it on a couple of machines, it is weird to me that this behaviour is not present on all my machines, and that this is happening!

Any insights?

Thanks!

2

I’m confused. If you haven’t put anything in the first block of your
sparse file, why is a problem if it is overwritten when you write to the
second block?

~Eric

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Friday, December 14, 2007 11:32 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] sparse file problem…

Hi all,

I’m currently having a strange problem with a sparse file.

I don’t know if this is normal or not, but here is my problem:

1- I create an empty sparse file of 200KB.
2- I write 200 bytes to the second block of the sparse file (64KB + some
offset)

The problem I then see is that the system overwrites the first block of
my sparse file, and fills it with 0…

If I write to the third block, I don’t see this problem, but every time
I do this on the second block, the first block is 0 filled…

Also, I only see this on one of my machines, which is 2003 server R2
sp2…

Although I’ve only tested it on a couple of machines, it is weird to me
that this behaviour is not present on all my machines, and that this is
happening!

Any insights?

Thanks!

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@edsiohio.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

3

I don’t see what you think is a problem.

The problem I then see is that the system overwrites the first block of my
sparse file, and fills it with 0…

Who cares? If it remains unallocated, reads will return zeroes. If it’s
allocated and zero filled, reads will return zeroes.

Also, I only see this on one of my machines, which is 2003 server R2 sp2…

Different cluster size?

You seem to be expecting that NTFS will use a certain “block” size for its
sparse files. But that is dependent on the volume cluster size and the
design decisions made by the NTFS devs. You shouldn’t care.

  • Dan.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Friday, December 14, 2007 9:32 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] sparse file problem…

Hi all,

I’m currently having a strange problem with a sparse file.

I don’t know if this is normal or not, but here is my problem:

1- I create an empty sparse file of 200KB.
2- I write 200 bytes to the second block of the sparse file (64KB + some
offset)

The problem I then see is that the system overwrites the first block of my
sparse file, and fills it with 0…

If I write to the third block, I don’t see this problem, but every time I do
this on the second block, the first block is 0 filled…

Also, I only see this on one of my machines, which is 2003 server R2 sp2…

Although I’ve only tested it on a couple of machines, it is weird to me that
this behaviour is not present on all my machines, and that this is
happening!

Any insights?

Thanks!

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@privtek.com To unsubscribe
send a blank email to xxxxx@lists.osr.com

4

If it is question of the size of things on the underlying filesystem,
try writing to the fourth block when the third is empty, and see if the
third gets overwritten.

~Eric

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Dan Kyler
Sent: Friday, December 14, 2007 11:51 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] sparse file problem…

I don’t see what you think is a problem.

The problem I then see is that the system overwrites the first block of

my
sparse file, and fills it with 0…

Who cares? If it remains unallocated, reads will return zeroes. If
it’s allocated and zero filled, reads will return zeroes.

Also, I only see this on one of my machines, which is 2003 server R2
sp2…

Different cluster size?

You seem to be expecting that NTFS will use a certain “block” size for
its sparse files. But that is dependent on the volume cluster size and
the design decisions made by the NTFS devs. You shouldn’t care.

  • Dan.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Friday, December 14, 2007 9:32 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] sparse file problem…

Hi all,

I’m currently having a strange problem with a sparse file.

I don’t know if this is normal or not, but here is my problem:

1- I create an empty sparse file of 200KB.
2- I write 200 bytes to the second block of the sparse file (64KB + some
offset)

The problem I then see is that the system overwrites the first block of
my sparse file, and fills it with 0…

If I write to the third block, I don’t see this problem, but every time
I do this on the second block, the first block is 0 filled…

Also, I only see this on one of my machines, which is 2003 server R2
sp2…

Although I’ve only tested it on a couple of machines, it is weird to me
that this behaviour is not present on all my machines, and that this is
happening!

Any insights?

Thanks!

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@privtek.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new
fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@edsiohio.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

5

The problem is that I am working on a HSM type system, where I actually have data offline for those sparse areas.

I have the sparse file local, with all the file marked as sparse. My application then writes to the second 64KB block of the file (64K + offset), and this causes my HSM code to download that second block, put it in the file, and then allow the write.

Once this write is finished, on some computers, I see the system write a pagedio write to the first block to fill it with 0’s…

when I then query the sparse API, it tells me that this portion of the file is now non sparse:

1- create sparse file with 200KB
2- fill with valid data
3- copy off data to somewhere
4- mark all areas as sparse

5- application writes to second 64KB block somewhere
6- I intercept this write, and fill the actual 64KB block with my valid data
7- I allow the write

8- eventually I see windows write to the first 64KB block, and 0 fill it

9- when I query the sparse API, I get back that the first block contains data

That is my current problem, since I then notify my application to get the new data for the first block, and it is no longer the same as the data I had…

6

Just as an additional info, I now see the difference between the computers, this happens on filesystems (NTFS) that have a 4KB cluster size. I’ve tried it on 512B cluster size, and it does not happen…

So, what happens is that I write to the second sparse block (sparse files seem to use 64KB blocks), and when I query the sparse API for the “in use” blocks, I get both the first and second 64KB block marked as in use, and I see a paged io for both areas from the system.

Anybody know if this is normal?

I know that in theory, a sparse file’s unused blocks are all read as “0” when read, but I thought that the sparse file API (FSCTL_QUERY_ALLOCATED_RANGES) would return only blocks that are now non sparse.

Thanks for any other insight.

7

Maybe I am misunderstanding how this works, and that is the base for this problem.

I’m trying to use the sparse API to see which blocks of the data are modified VS my original file.

i.e. I backup a file, and then set a reparse point, and an empty sparse file…

then when a user writes to the file, I use the sparse API to check which portions of the file have been modified.

It looks like from what I’m starting to see, that I cannot rely on this sparse API for this…

so another question: Can the file system mark any area of the file as sparse by itself after a user writes 0’s to the file, or is a FSCTL required to do so? (FSCTL_SET_ZERO_DATA)?? Or can the FileSystem by itself mark the area as sparse at some point?

Thanks!