LLVM port(s) take very long time to package

[Jan Beich]

Michael Gmelin <grembo at freebsd.org> writes:

>> On 12. Apr 2017, at 05:43, Alexey Dokuchaev <danfe at FreeBSD.org> wrote:
>> 
>> Hi there porters,
>> 
>> It had been annoying me for a while, so I figured I try asking here:
>> LLVM ports (particularly, llvm39-3.9.1) take abnormally long time to
>> package, e.g. right now it took 43 minutes (tmpfs-backed build area,
>> pkg-1.10.0_2.txz, AMD A8 APU @1900 MHz, plenty of RAM, no swap).
>> 
>> pkg was eating 100% (of a single core) during that time, so it looks
>> like it's actually busy (not waiting for I/O or something), but with
>> what?  And can it be fixed?  LLVM ports are real PITA to (re)build,
>> and while ccache helps to greatly reduce the build times, 3/4-hour
>> package time still cripples it.
>> 
>
> It's probably busy compressing the package. Try setting
> PKG_NOCOMPRESS=1 in /etc/make.conf (works ok when just building a
> package with make, didn't test with poudriere[0]). When creating
> packages, the current compression algorithm can't take advantage of
> multiple cores, so it's kind of a bottleneck on modern machines.

Nah, libarchive just defaults to 1 thread instead of using hw.ncpu.
The tricky part is propagating the fix into old releases or setting 
compression options from within pkg(8).

Index: lib/libarchive/config_freebsd.h
===================================================================
--- lib/libarchive/config_freebsd.h	(revision 316719)
+++ lib/libarchive/config_freebsd.h	(working copy)
@@ -139,6 +139,7 @@
 #define HAVE_LONG_LONG_INT 1
 #define HAVE_LSTAT 1
 #define HAVE_LUTIMES 1
+#define HAVE_LZMA_STREAM_ENCODER_MT 1
 #define HAVE_MBRTOWC 1
 #define HAVE_MEMMOVE 1
 #define HAVE_MEMORY_H 1
Index: contrib/libarchive/libarchive/archive_write_add_filter_xz.c
===================================================================
--- contrib/libarchive/libarchive/archive_write_add_filter_xz.c	(revision 316719)
+++ contrib/libarchive/libarchive/archive_write_add_filter_xz.c	(working copy)
@@ -152,7 +152,11 @@ common_setup(struct archive_write_filter *f)
 	}
 	f->data = data;
 	data->compression_level = LZMA_PRESET_DEFAULT;
+#ifdef HAVE_LZMA_STREAM_ENCODER_MT
+	data->threads = lzma_cputhreads();
+#else
 	data->threads = 1;
+#endif
 	f->open = &archive_compressor_xz_open;
 	f->close = archive_compressor_xz_close;
 	f->free = archive_compressor_xz_free;
Index: contrib/libarchive/libarchive/archive_write_set_format_xar.c
===================================================================
--- contrib/libarchive/libarchive/archive_write_set_format_xar.c	(revision 316719)
+++ contrib/libarchive/libarchive/archive_write_set_format_xar.c	(working copy)
@@ -384,7 +384,11 @@ archive_write_set_format_xar(struct archive *_a)
 	/* Set default compression type, level, and number of threads. */
 	xar->opt_compression = GZIP;
 	xar->opt_compression_level = 6;
+#ifdef HAVE_LZMA_STREAM_ENCODER_MT
+	xar->opt_threads = lzma_cputhreads();
+#else
 	xar->opt_threads = 1;
+#endif
 
 	a->format_data = xar;
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 584 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20170412/73f6f068/attachment.sig>