SMP Version of tar

[Yamagi Burmeister]

On Mon, 1 Oct 2012 22:16:53 -0700
Tim Kientzle <tim at kientzle.com> wrote:

> There are a few different parallel command-line compressors and decompressors in ports; experiment a lot (with large files being read from and/or written to disk) and see what the real effect is.  In particular, some decompression algorithms are actually faster than memcpy() when run on a single processor.  Parallelizing such algorithms is not likely to help much in the real world.
> 
> The two popular algorithms I would expect to benefit most are bzip2 compression and lzma compression (targeting xz or lzip format).  For decompression, bzip2 is block-oriented so fits SMP pretty naturally.  Other popular algorithms are stream-oriented and less amenable to parallelization.
> 
> Take a careful look at pbzip2, which is a parallelized bzip2/bunzip2 implementation that's already under a BSD license.  You should be able to get a lot of ideas about how to implement a parallel compression algorithm.  Better yet, you might be able to reuse a lot of the existing pbzip2 code.
> 
> Mark Adler's pigz is also worth studying.  It's also license-friendly, and is built on top of regular zlib, which is a nice technique when it's feasible.

Just a small note: There's a parallel implementation of xz called
"pixz". It's build atop of liblzma and libarchiv and stands under a 
BSD style license. See: https://github.com/vasi/pixz Maybe it's
possible to reuse most of the code.

-- 
Homepage:  www.yamagi.org
XMPP:      yamagi at yamagi.org
GnuPG/GPG: 0xEFBCCBCB
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20121002/72b0d664/attachment.pgp