ZFS/compression/performance

[Shivaram Upadhyayula]

On Thu, Oct 13, 2011 at 9:47 AM, Dennis Glatting <freebsd at penx.com> wrote:
> On Wed, 2011-10-12 at 08:59 +0100, Steven Hartland wrote:
> ----- Original Message -----
>>
>> From: "Dennis Glatting" <freebsd at penx.com>
>>
>>
<snip>
>> Have you tried using the alternative compression algorithms
>> e.g. lzjb or gzip-[1-5] the default gzip = gzip-6
>>
>
> I have tried lzjb and I am unimpressed. I have not tried different levels of
> gzip on ZFS but I have tried it on documents with results I expected.
>
> As I mentioned, I have a lot of data. Two files were 26GB uncompressed but I
> had to kill those data sets because I ran out of room (I have reorganized my
> arrays since then). My ZFS compression ratio is 4.93x and I would require
> more storage at different gzip levels or ljzb.
>
> An option is not too compress with ZFS rather directly with gzip however I
> would still need lots of temporary storage for manipulation, which is what I
> am doing now (e.g., sort). Processing with zcat isn't always a good solution
> because some applications want files, but you have to do what you have to
> do.
>

A few years back there was a discussion of about the possibility of
other compression algorithms in ZFS
(http://mail.opensolaris.org/pipermail/zfs-discuss/2007-June/011952.html).
But it looks like there hasn't been anything much further on that. I
have recently started using ZFS and during that time i have tried out
ZFS with lzf (http://oldhome.schmorp.de/marc/liblzf.html) and it seems
to perform much better, both in speed and ratio over lzjb.

Anyway my point is that, somewhere down the line other compression
algorithms should be evaluated. gzip seems slow and it looks like lzjb
may not be sufficient. For anyone interested, I have attached some of
the tests i had run and the diff for lzf support.

Cheers,
Shivaram

-- 
Reduce Storage expenditure with QUADStor Storage Virtualization
http://www.quadstor.com