git: ec2764d5ec49 - main - Mk/bsd.port.mk: prepare the land for pkg 1.17

[Stefan Esser]

Am 22.07.21 um 04:35 schrieb Matthew D. Fuller:
> On Thu, Jul 22, 2021 at 02:12:08AM +0000 I heard the voice of
> Alexey Dokuchaev, and lo! it spake thus:
>>
>> I've switched to PKG_NOCOMPRESS and
>> PKG_DEPENDS+=lbzip2:archivers/lbzip2 since 2017 when I got
>> dissatisfied with ridiculously long LLVM packaging times*
> 
> I've wished (not enough yet to see if I could manage it, but) to be
> able to use something like `gzip -1` for compression on some poudriere
> instances; all the clients pulling from it are on the same LAN if not
> the same physical machine, so bandwidth is trivially enough, and
> compression/decompression times eat up a significant amount of the
> cost of updates...

I have used .tar without compression on ZFS file systems that use zstd-2
compression (my default for all file systems not meant to hold already
compressed distfiles or media files).

This is the fastest and most efficient configuration IMHO (if you are
using ZFS anyway), with an effective compression ratio of more than 3
in practice and the compression time hidden in the filesystem layer.

If you are transferring these files over slow networks, that is not a
good setup, but within a system or on a LAN it does not matter.

Example of a randomly selected old package file of significant size:

$ cd /usr/ports/portmaster-backups
$ time cp llvm60-6.0.1_1.tar /tmp/

real	0m0.928s
user	0m0.000s
sys	0m0.930s

$ time cp /tmp/llvm60-6.0.1_1.tar .

real	0m0.572s
user	0m0.000s
sys	0m0.575s

$ ls -ls llvm60-6.0.1_1.tar /tmp/llvm60-6.0.1_1.tar
835724 -rw-r--r--  1 root  wheel  855781376 Jul 22 09:57 /tmp/llvm60-6.0.1_1.tar
290613 -rw-r--r--  1 root  wheel  855781376 Jul 22 09:58 llvm60-6.0.1_1.tar

The compression ratio is about 2.9, and the system time required for
reading the file (the copy to /tmp on tmpfs) is about doubled due
to the decompression performed by ZFS (compared to copying from a
ZFS file system without compression).

The compression time is not (fully) accounted to the process writing
the file to the compressed file system, but the real time is what
you observe from the point of view of the build process.

For comparison with user land compression

$ cd /tmp
$ time bzip2 -1 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.tbz

real	0m57.487s
user	0m57.318s
sys	0m0.169s

$ ls -ls llvm60-6.0.1_1.tbz
241348 -rw-r--r--  1 root  wheel  247136911 Jul 22 10:15 llvm60-6.0.1_1.tbz

$ time xz -1 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.txz

real	0m28.899s
user	0m28.714s
sys	0m0.183s

$ ls -ls llvm60-6.0.1_1.txz
191412 -rw-r--r--  1 root  wheel  196004988 Jul 22 10:17 llvm60-6.0.1_1.txz

$ time xz -9 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.txz

real	3m41.291s
user	3m40.973s
sys	0m0.314s

$ ls -ls /tmp/llvm60-6.0.1_1.txz
104196 -rw-r--r--  1 root  wheel  106694296 Jul 22 10:25 llvm60-6.0.1_1.txz

The compression achieved by using XZ is much higher than that of zstd-2,
but the time saving of using tar instead of txz is substantial, too.

I have just performed a test with zstd-19 for /usr/packages and found, that
the compression ratio is within 33% of that of xz -1 at much higher speed
(and nearly as good as bzip -1), and I'm considering to keep this setting
for /usr/packages:

$ ls -ls llvm60-6.0.1_1.tar
254307 -rw-r--r--  1 root  wheel  855781376 Jul 22 10:28 llvm60-6.0.1_1.tar

Regards, STefan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/dev-commits-ports-all/attachments/20210722/db61956c/attachment.sig>