[Bug 178504] [patch] switch logs compression from bzip2 to xz

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178504

David Chisnall <theraven at FreeBSD.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |theraven at FreeBSD.org

--- Comment #5 from David Chisnall <theraven at FreeBSD.org> ---
Do we have any measurements regarding the CPU / RAM overhead here?  Can this be
addressed by using a smaller block size without impacting compression ratios
too much?

A few measurements from my largest maillog file (852K uncompressed, much
smaller than on a large installation):

bzip2 baseline: 

7948  maximum resident set size
File size: 77KB.

xz --fast:

4328  maximum resident set size
File size: 80K

xz --best:

59848  maximum resident set size
File size: 70KB

xz -5:
26196  maximum resident set size
File size: 74KB

xz -block-size=1M:
26160  maximum resident set size
70K

The default appears to be --best.  

If I concatenate multiple rolled-over maillogs to give a 4.6MB file, I get this

bzip2 baseline: 

8316  maximum resident set size
File size: 443K.

xz --fast:

4436  maximum resident set size
File size: 447K

xz --best:

110092  maximum resident set size
File size: 382K

xz -5:
61524  maximum resident set size
File size: 412K

xz -block-size=1M

27920  maximum resident set size
File size: 396K

It looks as if setting the block size to 1MB prevents too much blowup in memory
for very large files, gives us better compression that bzip2, though at the
cost of more CPU and RAM.

I think that the correct solution for this is probably:

* Make the block size a configurable parameter
* Make the default 1M
* Make xz the default

This should let us get the benefit from better compression, but without letting
attackers consume too much memory (though they can consume CPU in proportion to
the size of the input file, with a larger constant multiplier for xz than
bzip2).

-- 
You are receiving this mail because:
You are the assignee for the bug.