SMP Version of tar

[John Nielsen]

On Oct 2, 2012, at 12:36 AM, Yamagi Burmeister <lists at yamagi.org> wrote:

> On Mon, 1 Oct 2012 22:16:53 -0700
> Tim Kientzle <tim at kientzle.com> wrote:
> 
>> There are a few different parallel command-line compressors and decompressors in ports; experiment a lot (with large files being read from and/or written to disk) and see what the real effect is.  In particular, some decompression algorithms are actually faster than memcpy() when run on a single processor.  Parallelizing such algorithms is not likely to help much in the real world.
>> 
>> The two popular algorithms I would expect to benefit most are bzip2 compression and lzma compression (targeting xz or lzip format).  For decompression, bzip2 is block-oriented so fits SMP pretty naturally.  Other popular algorithms are stream-oriented and less amenable to parallelization.
>> 
>> Take a careful look at pbzip2, which is a parallelized bzip2/bunzip2 implementation that's already under a BSD license.  You should be able to get a lot of ideas about how to implement a parallel compression algorithm.  Better yet, you might be able to reuse a lot of the existing pbzip2 code.
>> 
>> Mark Adler's pigz is also worth studying.  It's also license-friendly, and is built on top of regular zlib, which is a nice technique when it's feasible.
> 
> Just a small note: There's a parallel implementation of xz called
> "pixz". It's build atop of liblzma and libarchiv and stands under a 
> BSD style license. See: https://github.com/vasi/pixz Maybe it's
> possible to reuse most of the code.


See also below, which has some bugfixes/improvements that AFAIK were never committed in the original project (though they were submitted).
https://github.com/jlrobins/pixz

JN