SMP Version of tar

[Tim Kientzle]

On Oct 8, 2012, at 3:21 AM, Wojciech Puchar wrote:

>> Not necessarily.  If I understand correctly what Tim means, he's talking
>> about an in-memory compression of several blocks by several separate
>> threads, and then - after all the threads have compressed their
> 
> but gzip format is single stream. dictionary IMHO is not reset every X kilobytes.
> 
> parallel gzip is possible but not with same data format.

Yes, it is.

The following creates a compressed file that
is completely compatible with the standard
gzip/gunzip tools:

   * Break file into blocks
   * Compress each block into a gzip file (with gzip header and trailer information)
   * Concatenate the result.

This can be correctly decoded by gunzip.

In theory, you get slightly worse compression.  In practice, if your blocks are reasonably large (a megabyte or so each), the difference is negligible.

Tim