freebsd-fs Digest, Vol 437, Issue 3

Wed Nov 2 12:45:32 UTC 2011

On 2011-11-02 13:00, freebsd-fs-request at freebsd.org wrote:
> Message: 1
> Date: Tue, 1 Nov 2011 22:55:36 +0530
> From: Shivaram Upadhyayula<shivaram.u at quadstor.com>
> Subject: Re: ZFS/compression/performance
> To: Dennis Glatting<freebsd at penx.com>
> Cc:freebsd-fs at freebsd.org
> Message-ID:
> 	<CAN-_EfxVZchM-ByjbtMO2kcjiU_=u7GF5q5_9rz7HHeL09Vn0g at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Thu, Oct 13, 2011 at 9:47 AM, Dennis Glatting<freebsd at penx.com>  wrote:
>> >  On Wed, 2011-10-12 at 08:59 +0100, Steven Hartland wrote:
>> >  ----- Original Message -----
>>> >>
>>> >>  From: "Dennis Glatting"<freebsd at penx.com>
>>> >>
>>> >>
> <snip>
>>> >>  Have you tried using the alternative compression algorithms
>>> >>  e.g. lzjb or gzip-[1-5] the default gzip = gzip-6
>>> >>
>> >
>> >  I have tried lzjb and I am unimpressed. I have not tried different levels of
>> >  gzip on ZFS but I have tried it on documents with results I expected.
>> >
>> >  As I mentioned, I have a lot of data. Two files were 26GB uncompressed but I
>> >  had to kill those data sets because I ran out of room (I have reorganized my
>> >  arrays since then). My ZFS compression ratio is 4.93x and I would require
>> >  more storage at different gzip levels or ljzb.
>> >
>> >  An option is not too compress with ZFS rather directly with gzip however I
>> >  would still need lots of temporary storage for manipulation, which is what I
>> >  am doing now (e.g., sort). Processing with zcat isn't always a good solution
>> >  because some applications want files, but you have to do what you have to
>> >  do.
>> >
> A few years back there was a discussion of about the possibility of
> other compression algorithms in ZFS
> (http://mail.opensolaris.org/pipermail/zfs-discuss/2007-June/011952.html).
> But it looks like there hasn't been anything much further on that. I
> have recently started using ZFS and during that time i have tried out
> ZFS with lzf (http://oldhome.schmorp.de/marc/liblzf.html) and it seems
> to perform much better, both in speed and ratio over lzjb.
>
> Anyway my point is that, somewhere down the line other compression
> algorithms should be evaluated. gzip seems slow and it looks like lzjb
> may not be sufficient. For anyone interested, I have attached some of
> the tests i had run and the diff for lzf support.
>
> Cheers,
> Shivaram
>
> -- Reduce Storage expenditure with QUADStor Storage Virtualization 
> http://www.quadstor.com
I did some synthetic tests and lzf was OK, but not great. LZJB was 
terrible though.

http://encode.ru/threads/1266-In-memory-benchmark-with-fastest-LZSS-%28QuickLZ-Snappy%29-compressors?p=25913#post25913
https://extrememoderate.wordpress.com/2011/08/14/synthetic-test-of-filesystem-compression-part-1/
https://extrememoderate.wordpress.com/2011/08/20/synthetic-test-of-filesystem-compression-part-2/
I'm in the process of doing a more comprehensive part 3 and I think that 
LZ4 should be better.

-- 
Twoje radio