Sparse-file consistency

NTFSD
1

I’m testing following procedure:
1a) create a new file and mark it as sparse-file.
1b) set file’s size to (for example) 1000000 bytes.
1c) close this file.
2) open this file and write on position 100000[0x186a0] chracters
‘1234567890’ and close the file.
3) open this file again and write characters ‘1234567890’ on position
200000[0x30d40].
4) now I ask where are the allocated blocks (FSCTL_QUERY_ALLOCATED_RANGES)

And I get that allocated blocks (FSCTL_QUERY_ALLOCATED_RANGES) are here :
Offset Length
0x10000 0x10000
0x30000 0x10000

However when monitoring write operation at step 3 with FSpy I learned, that
write operations from Memory manager in fact fills with zeroes only part of
the file in the range 0x18000 - 0x20000 (0x8000 bytes).
How is this possible ? According to FSCTL_QUERY_ALLOCATED_RANGES the whole
range 0x10000-0x20000 (0x10000 bytes) should be filled with zeroes ?!

Thanks for hints.
VPR

2

Vladimir

So you have a sparse file where something has written (zeroes?) to the range
0x18000 - 0x20000 and you find that the range 0x10000-0x20000 is allocated.
So where is the surprise? You do understand spares files I hope? If you want
a range of a file to be unalllocated (perhaps this is what you mean when you
say filled with zeroes) you either must never have written or use
FSCTL_SET_ZERO_DATA. In fact if you do “fill with zeroes” by writing zeroes
then the filled bytes will be allocated! I believe this is all explained
well enough in the platfiorm sdk?

Good luck
Lyndon

“Vladimir Pribyl” wrote in message news:xxxxx@ntfsd…
> I’m testing following procedure:
> 1a) create a new file and mark it as sparse-file.
> 1b) set file’s size to (for example) 1000000 bytes.
> 1c) close this file.
> 2) open this file and write on position 100000[0x186a0] chracters
> ‘1234567890’ and close the file.
> 3) open this file again and write characters ‘1234567890’ on position
> 200000[0x30d40].
> 4) now I ask where are the allocated blocks (FSCTL_QUERY_ALLOCATED_RANGES)
>
> And I get that allocated blocks (FSCTL_QUERY_ALLOCATED_RANGES) are here :
> Offset Length
> 0x10000 0x10000
> 0x30000 0x10000
>
> However when monitoring write operation at step 3 with FSpy I learned,
> that write operations from Memory manager in fact fills with zeroes only
> part of the file in the range 0x18000 - 0x20000 (0x8000 bytes).
> How is this possible ? According to FSCTL_QUERY_ALLOCATED_RANGES the whole
> range 0x10000-0x20000 (0x10000 bytes) should be filled with zeroes ?!
>
> Thanks for hints.
> VPR
>
>

3

You are assuming that Mm is the OS component filling it with zeros, but
that’s not the case; look to the file system for the rest of the zero
filling.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vladimir Pribyl
Sent: Tuesday, August 09, 2005 11:14 AM
To: ntfsd redirect
Subject: [ntfsd] Sparse-file consistency

I’m testing following procedure:
1a) create a new file and mark it as sparse-file.
1b) set file’s size to (for example) 1000000 bytes.
1c) close this file.
2) open this file and write on position 100000[0x186a0] chracters
‘1234567890’ and close the file.
3) open this file again and write characters ‘1234567890’ on position
200000[0x30d40].
4) now I ask where are the allocated blocks
(FSCTL_QUERY_ALLOCATED_RANGES)

And I get that allocated blocks (FSCTL_QUERY_ALLOCATED_RANGES) are here
:
Offset Length
0x10000 0x10000
0x30000 0x10000

However when monitoring write operation at step 3 with FSpy I learned,
that
write operations from Memory manager in fact fills with zeroes only part
of
the file in the range 0x18000 - 0x20000 (0x8000 bytes).
How is this possible ? According to FSCTL_QUERY_ALLOCATED_RANGES the
whole
range 0x10000-0x20000 (0x10000 bytes) should be filled with zeroes ?!

Thanks for hints.
VPR

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

4

> write operations from Memory manager in fact fills with zeroes only part of

the file in the range 0x18000 - 0x20000 (0x8000 bytes).
How is this possible ? According to FSCTL_QUERY_ALLOCATED_RANGES

So what? Maybe NTFS itself zeroes the rest?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

5

Thanks for reply. But how can I work with this write (0x10000-0x18000), when
I don’t see it. I need to encrypt these zeroes too, so how should I do it,
if I don’t see this noncached write ?
VPR

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntfsd…
>> write operations from Memory manager in fact fills with zeroes only part
>> of
>> the file in the range 0x18000 - 0x20000 (0x8000 bytes).
>> How is this possible ? According to FSCTL_QUERY_ALLOCATED_RANGES
>
> So what? Maybe NTFS itself zeroes the rest?
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>

6

First, may be better to ask the good question :
Is it possible to do a cyphering filter driver working whith sparse file ?
I think it is not.

Vladimir Pribyl wrote:

Thanks for reply. But how can I work with this write (0x10000-0x18000), when
I don’t see it. I need to encrypt these zeroes too, so how should I do it,
if I don’t see this noncached write ?
VPR

“Maxim S. Shatskih” wrote in message
> news:xxxxx@ntfsd…
>
>>>write operations from Memory manager in fact fills with zeroes only part
>>>of
>>>the file in the range 0x18000 - 0x20000 (0x8000 bytes).
>>>How is this possible ? According to FSCTL_QUERY_ALLOCATED_RANGES
>>
>>So what? Maybe NTFS itself zeroes the rest?
>>
>>Maxim Shatskih, Windows DDK MVP
>>StorageCraft Corporation
>>xxxxx@storagecraft.com
>>http://www.storagecraft.com
>>
>>
>
>
>
>

7

Of course it is possible. However, the question should be: what is the
point of encrypting large blocks of zeros - which creates large blocks
of random data that is NOT zero?

By the time you are done, you won’t benefit from the sparse nature of
the file at all.

So, while possible, it seems a tremendous amount of effort to implement
something that won’t be useful. The same thing applies to supporting
compressed files in NTFS with an encryption filter above - you can
eventually make it work but when you’re done it is pointless because
your encrypted data has so little redundancy that the compression is
useless. So you’ve done a lot of work and gained nothing in return.

What do you think you’ll gain by marking an encrypted file as sparse?
Does your encryption algorithm generate large empty (sparse) fields?

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of insakda
Sent: Wednesday, August 10, 2005 10:12 AM
To: ntfsd redirect
Subject: Re:[ntfsd] Sparse-file consistency

First, may be better to ask the good question :
Is it possible to do a cyphering filter driver working whith sparse file
?
I think it is not.

Vladimir Pribyl wrote:

Thanks for reply. But how can I work with this write
(0x10000-0x18000), when
I don’t see it. I need to encrypt these zeroes too, so how should I do
it,
if I don’t see this noncached write ?
VPR

“Maxim S. Shatskih” wrote in message
> news:xxxxx@ntfsd…
>
>>>write operations from Memory manager in fact fills with zeroes only
part
>>>of
>>>the file in the range 0x18000 - 0x20000 (0x8000 bytes).
>>>How is this possible ? According to FSCTL_QUERY_ALLOCATED_RANGES
>>
>>So what? Maybe NTFS itself zeroes the rest?
>>
>>Maxim Shatskih, Windows DDK MVP
>>StorageCraft Corporation
>>xxxxx@storagecraft.com
>>http://www.storagecraft.com
>>
>>
>
>
>
>

—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

8

> > I don’t see it. I need to encrypt these zeroes too, so how should I do it,

Impossible with a sparse file. Either you have the zeros as sparse - i.e. not
crypted, or you prohibit the sparse file creation on your filter.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

9

True, yet, NTFS does allow a file to be both sparse and encrypted, at least
in XPSP2 :slight_smile:

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntfsd…
>> > I don’t see it. I need to encrypt these zeroes too, so how should I do
>> > it,
>
> Impossible with a sparse file. Either you have the zeros as sparse - i.e.
> not
> crypted, or you prohibit the sparse file creation on your filter.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>