svn commit: r357609 - in head/contrib/xz/src: common xz

[Xin LI]

Author: delphij
Date: Thu Feb  6 07:47:28 2020
New Revision: 357609
URL: https://svnweb.freebsd.org/changeset/base/357609

Log:
  MFV r357608: Limit memory usage in xz(1) instead of in tuklib.
  
  Apply upstream 353970510895f6a80adfe60cf71b70a95adfa8bc to limit memory
  usage on 32-bit binary to 4020 MiB.
  
  Submitted by:	Lasse Collin <lasse.collin at tukaani.org>
  Reviewed by:	kib, bcr
  Differential Revision:	https://reviews.freebsd.org/D23474

Modified:
  head/contrib/xz/src/common/tuklib_physmem.c
  head/contrib/xz/src/xz/hardware.c
  head/contrib/xz/src/xz/xz.1
Directory Properties:
  head/contrib/xz/   (props changed)

Modified: head/contrib/xz/src/common/tuklib_physmem.c
==============================================================================
--- head/contrib/xz/src/common/tuklib_physmem.c	Thu Feb  6 07:45:07 2020	(r357608)
+++ head/contrib/xz/src/common/tuklib_physmem.c	Thu Feb  6 07:47:28 2020	(r357609)
@@ -45,7 +45,6 @@
 #	include <sys/systemcfg.h>
 
 #elif defined(TUKLIB_PHYSMEM_SYSCONF)
-#	include <limits.h>
 #	include <unistd.h>
 
 #elif defined(TUKLIB_PHYSMEM_SYSCTL)
@@ -146,16 +145,13 @@ tuklib_physmem(void)
 #elif defined(TUKLIB_PHYSMEM_SYSCONF)
 	const long pagesize = sysconf(_SC_PAGESIZE);
 	const long pages = sysconf(_SC_PHYS_PAGES);
-	if (pagesize != -1 && pages != -1) {
+	if (pagesize != -1 && pages != -1)
 		// According to docs, pagesize * pages can overflow.
 		// Simple case is 32-bit box with 4 GiB or more RAM,
 		// which may report exactly 4 GiB of RAM, and "long"
 		// being 32-bit will overflow. Casting to uint64_t
 		// hopefully avoids overflows in the near future.
 		ret = (uint64_t)pagesize * (uint64_t)pages;
-		if (ret > SIZE_T_MAX)
-			ret = SIZE_T_MAX;
-	}
 
 #elif defined(TUKLIB_PHYSMEM_SYSCTL)
 	int name[2] = {

Modified: head/contrib/xz/src/xz/hardware.c
==============================================================================
--- head/contrib/xz/src/xz/hardware.c	Thu Feb  6 07:45:07 2020	(r357608)
+++ head/contrib/xz/src/xz/hardware.c	Thu Feb  6 07:47:28 2020	(r357609)
@@ -68,8 +68,38 @@ hardware_memlimit_set(uint64_t new_memlimit,
 		new_memlimit = (uint32_t)new_memlimit * total_ram / 100;
 	}
 
-	if (set_compress)
+	if (set_compress) {
 		memlimit_compress = new_memlimit;
+
+#if SIZE_MAX == UINT32_MAX
+		// FIXME?
+		//
+		// When running a 32-bit xz on a system with a lot of RAM and
+		// using a percentage-based memory limit, the result can be
+		// bigger than the 32-bit address space. Limiting the limit
+		// below SIZE_MAX for compression (not decompression) makes
+		// xz lower the compression settings (or number of threads)
+		// to a level that *might* work. In practice it has worked
+		// when using a 64-bit kernel that gives full 4 GiB address
+		// space to 32-bit programs. In other situations this might
+		// still be too high, like 32-bit kernels that may give much
+		// less than 4 GiB to a single application.
+		//
+		// So this is an ugly hack but I will keep it here while
+		// it does more good than bad.
+		//
+		// Use a value less than SIZE_MAX so that there's some room
+		// for the xz program and so on. Don't use 4000 MiB because
+		// it could look like someone mixed up base-2 and base-10.
+		const uint64_t limit_max = UINT64_C(4020) << 20;
+
+		// UINT64_MAX is a special case for the string "max" so
+		// that has to be handled specially.
+		if (memlimit_compress != UINT64_MAX
+				&& memlimit_compress > limit_max)
+			memlimit_compress = limit_max;
+#endif
+	}
 
 	if (set_decompress)
 		memlimit_decompress = new_memlimit;

Modified: head/contrib/xz/src/xz/xz.1
==============================================================================
--- head/contrib/xz/src/xz/xz.1	Thu Feb  6 07:45:07 2020	(r357608)
+++ head/contrib/xz/src/xz/xz.1	Thu Feb  6 07:47:28 2020	(r357609)
@@ -1005,6 +1005,25 @@ instead of
 until the details have been decided.
 .RE
 .IP ""
+For 32-bit
+.BR xz
+there is a special case: if the
+.I limit
+would be over
+.BR "4020\ MiB" ,
+the
+.I limit
+is set to
+.BR "4020\ MiB" .
+(The values
+.B 0
+and
+.B max
+aren't affected by this.
+A similar feature doesn't exist for decompression.)
+This can be helpful when a 32-bit executable has access
+to 4\ GiB address space while hopefully doing no harm in other situations.
+.IP ""
 See also the section
 .BR "Memory usage" .
 .TP