svn commit: r355004 - head/sys/vm

[Mark Johnston]

Author: markj
Date: Fri Nov 22 16:31:30 2019
New Revision: 355004
URL: https://svnweb.freebsd.org/changeset/base/355004

Log:
  Reclaim memory from UMA if the page daemon is struggling.
  
  Use the UMA reclaim thread to asynchronously drain all caches if
  there is a severe shortage in a domain.  Otherwise we only trigger UMA
  reclamation every 10s even when the system has completely run out of
  memory.
  
  Stop entirely draining the caches when one domain falls below its min
  threshold.  In some workloads it is normal for one NUMA domain to end
  up being nearly depleted by kernel memory allocations, for example for
  the ZFS ARC.  The domainset iterators skip domains below the
  vmd_min_free theshold on the first iteration, so we should allow that
  mechanism to limit further depletion of the domain's free pages before
  taking the extreme step of calling uma_reclaim(UMA_RECLAIM_DRAIN_CPU).
  
  Discussed with:	jeff
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D22395

Modified:
  head/sys/vm/vm_pageout.c

Modified: head/sys/vm/vm_pageout.c
==============================================================================
--- head/sys/vm/vm_pageout.c	Fri Nov 22 16:31:10 2019	(r355003)
+++ head/sys/vm/vm_pageout.c	Fri Nov 22 16:31:30 2019	(r355004)
@@ -1965,12 +1965,20 @@ vm_pageout_oom(int shortage)
 	}
 }
 
+/*
+ * Signal a free page shortage to subsystems that have registered an event
+ * handler.  Reclaim memory from UMA in the event of a severe shortage.
+ * Return true if the free page count should be re-evaluated.
+ */
 static bool
 vm_pageout_lowmem(void)
 {
 	static int lowmem_ticks = 0;
 	int last;
+	bool ret;
 
+	ret = false;
+
 	last = atomic_load_int(&lowmem_ticks);
 	while ((u_int)(ticks - last) / hz >= lowmem_period) {
 		if (atomic_fcmpset_int(&lowmem_ticks, &last, ticks) == 0)
@@ -1984,15 +1992,27 @@ vm_pageout_lowmem(void)
 
 		/*
 		 * We do this explicitly after the caches have been
-		 * drained above.  If we have a severe page shortage on
-		 * our hands, completely drain all UMA zones.  Otherwise,
-		 * just prune the caches.
+		 * drained above.
 		 */
-		uma_reclaim(vm_page_count_min() ? UMA_RECLAIM_DRAIN_CPU :
-		    UMA_RECLAIM_TRIM);
-		return (true);
+		uma_reclaim(UMA_RECLAIM_TRIM);
+		ret = true;
 	}
-	return (false);
+
+	/*
+	 * Kick off an asynchronous reclaim of cached memory if one of the
+	 * page daemons is failing to keep up with demand.  Use the "severe"
+	 * threshold instead of "min" to ensure that we do not blow away the
+	 * caches if a subset of the NUMA domains are depleted by kernel memory
+	 * allocations; the domainset iterators automatically skip domains
+	 * below the "min" threshold on the first pass.
+	 *
+	 * UMA reclaim worker has its own rate-limiting mechanism, so don't
+	 * worry about kicking it too often.
+	 */
+	if (vm_page_count_severe())
+		uma_reclaim_wakeup();
+
+	return (ret);
 }
 
 static void