svn commit: r366839 - head/sys/vm

[Mark Johnston]

Author: markj
Date: Mon Oct 19 16:55:03 2020
New Revision: 366839
URL: https://svnweb.freebsd.org/changeset/base/366839

Log:
  uma: Avoid depleting keg reserves when filling a bucket
  
  zone_import() fetches a free or partially free slab from the keg and
  then uses its items to populate an array, typically filling a bucket.
  If a single allocation causes the keg to drop below its minimum reserve,
  the inner loop ends.  However, if the bucket is still not full and
  M_USE_RESERVE is specified, the outer loop will continue to fetch items
  from the keg.
  
  If M_USE_RESERVE is specified and the number of free items is below the
  reserved limit, we should return only a single item.  Otherwise, if the
  bucket size is larger than the reserve, all of the reserved items may
  end up in a single per-CPU bucket, invisible to other CPUs.
  
  Reviewed by:	rlibby
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D26771

Modified:
  head/sys/vm/uma_core.c

Modified: head/sys/vm/uma_core.c
==============================================================================
--- head/sys/vm/uma_core.c	Mon Oct 19 16:54:06 2020	(r366838)
+++ head/sys/vm/uma_core.c	Mon Oct 19 16:55:03 2020	(r366839)
@@ -3734,10 +3734,17 @@ zone_import(void *arg, void **bucket, int max, int dom
 		stripe = howmany(max, vm_ndomains);
 #endif
 		dom = &keg->uk_domain[slab->us_domain];
-		while (slab->us_freecount && i < max) { 
+		do {
 			bucket[i++] = slab_alloc_item(keg, slab);
-			if (dom->ud_free_items <= keg->uk_reserve)
-				break;
+			if (dom->ud_free_items <= keg->uk_reserve) {
+				/*
+				 * Avoid depleting the reserve after a
+				 * successful item allocation, even if
+				 * M_USE_RESERVE is specified.
+				 */
+				KEG_UNLOCK(keg, slab->us_domain);
+				goto out;
+			}
 #ifdef NUMA
 			/*
 			 * If the zone is striped we pick a new slab for every
@@ -3751,13 +3758,14 @@ zone_import(void *arg, void **bucket, int max, int dom
 			    vm_ndomains > 1 && --stripe == 0)
 				break;
 #endif
-		}
+		} while (slab->us_freecount != 0 && i < max);
 		KEG_UNLOCK(keg, slab->us_domain);
+
 		/* Don't block if we allocated any successfully. */
 		flags &= ~M_WAITOK;
 		flags |= M_NOWAIT;
 	}
-
+out:
 	return i;
 }