RtlCompressBuffer compression ratio

NTFSD
1

Using RtlCompressBuffer I have not been able to come anywhere close to the
size reduction that Windows gets when compressing a file from Explorer. I
assume that the OS uses the same API. I have tried all different variations
on chunk size and buffer size, but I always end up with at least twice the
size as a zip folder.

I am using these flags:

COMPRESSION_FORMAT_LZNT1 | COMPRESSION_ENGINE_MAXIMUM

Even tried COMPRESSION_ENGINE_HIBER, but that just crashed.

Are there some undocumented parameters to RtlCompressBuffer or some
undocumented APIs?

Thanks.

2

On 4/30/2010 2:18 PM, Neil Weicher wrote:

Using RtlCompressBuffer I have not been able to come anywhere close to the
size reduction that Windows gets when compressing a file from Explorer. I
assume that the OS uses the same API. I have tried all different variations
on chunk size and buffer size, but I always end up with at least twice the
size as a zip folder.

As far as I have understood it, zip has its own compression algorithm
which was licensed to MSFT. The Rtl library uses a different compression
algorithm altogether. You can see this by setting a bp on the Rtl call
and then zip up some files, you won’t hit the bp.

Pete

–
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295

3

I don’t know much about RtlCompressBuffer(), but based on the difference in sizes, I would guess that Explorer is probably using LZX compression.

You can experiment with this out one the command line with CABARC:

cabarc -m MSZIP …
cabarc -m LZX …

mm

4

RtlCompressBuffer() implements LZNT1 compression. It’s fast enough and good enough, though not great. You can only compress up to 64K at a time (hardcoded limit), which really limits the overall compression ratio you can achieve when compressing a large stream. This is the compression routine that NTFS uses for its compression feature. NTFS compresses a file in units of up to 64K at a time, which jives nicely with the hardcoded limit in RtlCompressBuffer().

Explorer’s “Send to compressed (zipped) folder” feature simply creates a .zip file, using the public domain ZIP format.

CABARC creates .cab files, which use the Cabinet format. Cabinet is a proprietary Microsoft format which is most definitely not the same as the ZIP format. The codec options (MSZIP, LZX) are also proprietary Microsoft formats and not the same as those used in ZIP. MSZIP is based on Huffman, while LZX is based on Lempel-Ziv and is generally much better than MSZIP. Note that Cabinet takes advantage of cross-file compression (giving Cabinet a compression advantage over ZIP), whereas ZIP compresses each file individually (giving ZIP a speed advantage when extracting a single file).

Hope this clarifies things a bit.

Craig

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-409962-
xxxxx@lists.osr.com] On Behalf Of xxxxx@evitechnology.com
Sent: Friday, April 30, 2010 1:53 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] RtlCompressBuffer compression ratio

I don’t know much about RtlCompressBuffer(), but based on the difference
in sizes, I would guess that Explorer is probably using LZX compression.

You can experiment with this out one the command line with CABARC:

cabarc -m MSZIP …
cabarc -m LZX …

mm

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars (including our new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

5

> Using RtlCompressBuffer

It is used for “blue” NTFS compressed files, not for shell’s Vista+ compressed folders.

The latter are good old ZIP files.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

6

My bad.

mm

7

Thanks for the info. Should have realized that they were using different
algorithms!

“Craig Barkhouse” wrote in message
news:xxxxx@ntfsd…
RtlCompressBuffer() implements LZNT1 compression. It’s fast enough and good
enough, though not great. You can only compress up to 64K at a time
(hardcoded limit), which really limits the overall compression ratio you can
achieve when compressing a large stream. This is the compression routine
that NTFS uses for its compression feature. NTFS compresses a file in units
of up to 64K at a time, which jives nicely with the hardcoded limit in
RtlCompressBuffer().

Explorer’s “Send to compressed (zipped) folder” feature simply creates a
.zip file, using the public domain ZIP format.

CABARC creates .cab files, which use the Cabinet format. Cabinet is a
proprietary Microsoft format which is most definitely not the same as the
ZIP format. The codec options (MSZIP, LZX) are also proprietary Microsoft
formats and not the same as those used in ZIP. MSZIP is based on Huffman,
while LZX is based on Lempel-Ziv and is generally much better than MSZIP.
Note that Cabinet takes advantage of cross-file compression (giving Cabinet
a compression advantage over ZIP), whereas ZIP compresses each file
individually (giving ZIP a speed advantage when extracting a single file).

Hope this clarifies things a bit.

Craig

> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:bounce-409962-
> xxxxx@lists.osr.com] On Behalf Of xxxxx@evitechnology.com
> Sent: Friday, April 30, 2010 1:53 PM
> To: Windows File Systems Devs Interest List
> Subject: RE:[ntfsd] RtlCompressBuffer compression ratio
>
> I don’t know much about RtlCompressBuffer(), but based on the difference
> in sizes, I would guess that Explorer is probably using LZX compression.
>
> You can experiment with this out one the command line with CABARC:
>
> cabarc -m MSZIP …
> cabarc -m LZX …
>
>
> mm
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars (including our new
> fs
> mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

8

“Peter Scott” wrote in message
news:xxxxx@ntfsd…
> On 4/30/2010 2:18 PM, Neil Weicher wrote:
>> Using RtlCompressBuffer I have not been able to come anywhere close to
>> the
>> size reduction that Windows gets when compressing a file from Explorer. I
>> assume that the OS uses the same API. I have tried all different
>> variations
>> on chunk size and buffer size, but I always end up with at least twice
>> the
>> size as a zip folder.
>
> As far as I have understood it, zip has its own compression algorithm
> which was licensed to MSFT. The Rtl library uses a different compression
> algorithm altogether. You can see this by setting a bp on the Rtl call and
> then zip up some files, you won’t hit the bp.
>

Peter,

Makes perfect sense now. Did the following test with an 76.5MB file:

NTFS compression: 20.9mb (27%)
Zip maximum: 7.75mb (10%)
7Zip LZMA maximum: 2.99mb (4%)

Hmmm… anyone have the source code for the equivalent of RtlCompressBuffer
for LZMA?

Neil

9

> Hmmm… anyone have the source code for the equivalent of RtlCompressBuffer

for LZMA?

google:7zip sdk -> http://www.7-zip.org/sdk.html

t.

Neil

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

10

Indeed; there’s a ton of linux packages that use lzma - LZMA, XZ Utils, et. c.

Good luck,

mm