FreeBSD ports USE_XZ critical issue on low-RAM computers

[Matthias Andree]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

(blind carbon copy to Lasse Collin, maintainer of xzutils upstream)

Greetings,

I've just had xz break my devel/libtool22 FreeBSD port build on a low
memory computer (128 MB).

Reason is that xz by default caps memory use at ~40% of physical RAM
(this is documented), and skips decompressing files if that doesn't
suffice. Options to override this are provided, but I question this
behaviour:

- - This xz feature may seem reasonable if we do have a choice about
memory use, meaning, during compression. There we can reduce dictionary
size, search depth, and all that, and reduce memory use at the expense
of compression factor.  No big deal.

- - This feature is, in my perception, INADEQUATE during decompression. If
I have a .xz file (downloaded from the Internet) that needs 90 MB RAM to
decompress, then I need to use those 90 MB no matter if that's nice or
not, it's just critical.

I am proposing to Lasse Collin to drop memory capping functionality in
xz/lzma in decompress mode, and in lzmadec/xzdec.


This is what happens in practice:

> # cd /usr/ports/devel/libtool22
> # make
> ===>  Vulnerability check disabled, database not found
> ===>  License check disabled, port has not defined LICENSE
> ===>  Extracting for libtool-2.2.6b
> => MD5 Checksum OK for libtool-2.2.6b.tar.lzma.
> => SHA256 Checksum OK for libtool-2.2.6b.tar.lzma.
> ===>   libtool-2.2.6b depends on file: /usr/local/bin/xz - found
> /usr/local/bin/xz: /usr/ports/distfiles//libtool-2.2.6b.tar.lzma: Memory usage limit reached
> /usr/local/bin/xz: Limit was 46 MiB, but 65 MiB would have been needed
> ===>  Patching for libtool-2.2.6b
> ===>  Applying FreeBSD patches for libtool-2.2.6b
> patch: **** can't cd to /usr/ports/devel/libtool22/work/libtool-2.2.6b: No such file or directory
> => Patch patch-libltdl-Makefile.in failed to apply cleanly.
> *** Error code 1
> 
> Stop in /usr/ports/devel/libtool22.

Investigating this (on FreeBSD 6.4-RELEASE-p10 i386):

- - The ports system generates this command line and executes it:

> for file in libtool-2.2.6b.tar.lzma; do  if ! (cd /usr/ports/devel/libtool22/work && /usr/local/bin/xz -dc /usr/ports/distfiles//$file | /usr/bin/tar -xf - --no-same-owner --no-same-permissions); then  exit 1;  fi;  done

- - xz fails with error message as above, exits with return code 1, after
not having printed any bytes on stdout.

- - tar decompresses a 0-byte file, which succeeds.

- - The FreeBSD ports system does not detect the tar input was empty, and
proceeds to the patch phase, which fails because tar didn't extract
files - so patch fails. This misleads end users about the actual error.


2. xz port or its use in the framework: we need to disable the memory
capping for decompressing. We simply must try to decompress no matter what.

I'll wait a couple of days hoping for Lasse's response, but generally,
we have two alternatives to proceed:

a. directly patch archivers/xz so that the memory capping is not applied
during decompression. Should be considered upstream -- because during
decompression you cannot tweak parameters to reduce memory usage, you
need to proceed with what's needed by the file to build the dictionaries
and whatnot to decompress. (CC'ing port maintainer Christian Weisgerber)

b. ports framework: pass command line arguments or environment variables
to defeat capping with the -M option. Note that -M 0 sets the default
cap to 40% of physical RAM. Example (the large figure is 2^31-1 i. e.
INT_MAX):

> # export XZ_OPT=-M2147483647
> # xz -dc >/dev/null /usr/ports/distfiles/libtool*.tar.lzma
> # echo $?
> 0

I'd propose using XT_OPT+=-M$(getconf LLONG_MAX) for now.

We can't use unsigned types because FreeBSD getconf, as of 6.4, would
return those as negative numbers (due to internal use of intmax_t even
where variables are of unsigned type and might be in range (INTMAX_MAX,
UINTMAX_MAX], which causes (justified!) xz complaints, and INTMAX_MAX or
INT64_MAX cannot be queried via getconf(1) on FreeBSD (undefined).

The getconf fix is non-trivial, so I expect we better not rely on that now.

So for FreeBSD, I propose to patch this Mk/bsd.port.mk line #2382 (as of
CVS rev. 1.642)

EXTRACT_CMD?=           ${XZ_CMD}

to

EXTRACT_CMD?=           ${XZ_CMD} $$(getconf LLONG_MAX)


Other proposals?

Best regards
Matthias

- -- 
Matthias Andree (ports/ committer)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.12 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iEYEARECAAYFAkwbpNMACgkQvmGDOQUufZXWJQCgmSOmo5a0rnFR3aNa7/4NEwGk
pQ8AoJ2liNx8o62j5Z6ON1Lh22n60hia
=pYpJ
-----END PGP SIGNATURE-----