From nobody Mon Aug 24 16:30:00 2026 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hTGbY1RlLz6pbYs for ; Mon, 24 Aug 2026 16:30:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hTGbX1nl3z41KB for ; Mon, 24 Aug 2026 16:30:00 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787589000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=u8M/LU5N7mwdloacGL0P5xg6n9iAyPGktRbgGeS54rg=; b=nn/D1CZbPSIKFsfSj/K1A2bVxYWO+6r53h23cbBCdhfQZq78Q7hi7JR41dBEI1rsV3pkGO mIindiYJPCsqTU/MmZ1ouyN2JuAcc2AmLUFcYfduoyzjcLBd54Tt8PeWausGBNzhUca/+G OQEWyO1EgIVkjV2zdDbiKobTNzO9WjIhn+tnV+GysU2RQ1nMQ5+PxG2QeARWlsfc0VBQdH i6hJoc2Kq7TeJAlF8k9SBJ4p3yXwzYwyKtcV4shFkyi2t6lCy+XTHzWyX0L7P10ojc3Fce Oo7OJCHmEb678BuW/x77OkDghvUUiBrIsSK+VnoCP5BhQ4sVzjnBU2jTFwEYeA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1787589000; a=rsa-sha256; cv=none; b=Nj4U0XDzTsB6gW6UOTqfEEenVhi/0nf36oI7gbMQ0CGK9kBaTzDmH2+Pfsv44nCtEnXVAs tXLvqQNZPM6OyEhR+GCN+YBzzeLlsogILpFJDCKwrqbxc/MgSN57Li1MbvGJ8X78mT+BBb nNuh6xlvMWcBQBpHJtepVZNLusWCnCRLN8jd8Eb8OPknbm3MbsseMUlrgvz2gGmjzdCBpX GooaSpcEL7UjMuhR4Cc2f37O/oZFHCZIidibC9Sa+fn82UZCMJlBB22zU1pdnCy9g6VCcw 0oTpaSt0Gm7IwDFpX+nFMtUBy7tKy8pp9wo3uiVEssbnwOVana0tsh3Jt00XWA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787589000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=u8M/LU5N7mwdloacGL0P5xg6n9iAyPGktRbgGeS54rg=; b=xZ+0vRwMsT26cT+FklqD9xkP+NvF+P29j/kTynB/mUnK91CUG0ObgDNzKZ81SzRHBuR4aU Fh+rY908mV1UQTkY0cZXX5PRmA70D2AW5c0UMMuHpvEOxxV729I/tQo9w10HmeqBcgLkYB tK8IiDNnXLgn+rF3V6el8LMLQo3wVDMs3NhspogQImEVkbyilGmHlUHSrqpMQ98cuhzjFm 6IcpqMhz798RLjDZghFXIWx8UMjFFLnO6Q2P7sh7BY9xRJ1UXOPo/QIQPFiIuPValA8mwz Mp6tbDQx8XUBNzNVzFL9ofSZxjOw5t/UW+mKNSviexd9QS5qGHTVj5+NLYKrpA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hTGbX0QFWz1BDR for ; Mon, 24 Aug 2026 16:30:00 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 2250e by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Mon, 24 Aug 2026 16:30:00 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 4b94ab9b7c5e - stable/15 - uma: Factor out the implementations of uma_zfree_{arg,smr}() List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 4b94ab9b7c5e30cd650c63c029eaac25e3908c18 Auto-Submitted: auto-generated Date: Mon, 24 Aug 2026 16:30:00 +0000 Message-Id: <6a8c7188.2250e.65f7ca9b@gitrepo.freebsd.org> The branch stable/15 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=4b94ab9b7c5e30cd650c63c029eaac25e3908c18 commit 4b94ab9b7c5e30cd650c63c029eaac25e3908c18 Author: Mark Johnston AuthorDate: 2026-07-24 21:11:58 +0000 Commit: Mark Johnston CommitDate: 2026-08-24 15:17:54 +0000 uma: Factor out the implementations of uma_zfree_{arg,smr}() The two function both free an item to a UMA zone, but uma_zfree_arg() does so in such as way as to ensure that the item will be the first one returned by a subsequent allocation, while uma_zfree_smr() must defer reuse of the item and therefore never frees to the per-CPU alloc bucket. When KASAN is enabled, we actually want uma_zfree_arg() to behave like uma_zfree_smr(): to improve the reliability of use-after-free detection, reuse of the newly freed item should be deferred for some time. Refactor a bit to make it easier to improve KASAN along these lines: introduce two helper functions, cache_free_item() and cache_free_smr(), which handle most of the work of interacting with the per-CPU caches. A subsequent commit will let uma_zfree_arg() use cache_free_smr() when KASAN is enabled. No functional change intended. Reviewed by: rlibby MFC after: 1 month Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D58268 (cherry picked from commit 6337ca19a3637aa72eddf4d62a2eaf7d8df51638) --- sys/vm/uma_core.c | 159 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 91 insertions(+), 68 deletions(-) diff --git a/sys/vm/uma_core.c b/sys/vm/uma_core.c index 5e63659ec06c..f5500e507ccc 100644 --- a/sys/vm/uma_core.c +++ b/sys/vm/uma_core.c @@ -4461,37 +4461,82 @@ fail: return (NULL); } -/* See uma.h */ -void -uma_zfree_smr(uma_zone_t zone, void *item) +static __always_inline bool +cache_free_item(uma_zone_t zone, int uz_flags, void *item, void *udata) { uma_cache_t cache; - uma_cache_bucket_t bucket; int itemdomain; + + /* + * If possible, free to the per-CPU cache. There are two + * requirements for safe access to the per-CPU cache: (1) the thread + * accessing the cache must not be preempted or yield during access, + * and (2) the thread must not migrate CPUs without switching which + * cache it accesses. We rely on a critical section to prevent + * preemption and migration. We release the critical section in + * order to acquire the zone mutex if we are unable to free to the + * current cache; when we re-acquire the critical section, we must + * detect and handle migration if it has occurred. + */ + itemdomain = 0; #ifdef NUMA - int uz_flags; + if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0) + itemdomain = item_domain(item); #endif - CTR3(KTR_UMA, "uma_zfree_smr zone %s(%p) item %p", - zone->uz_name, zone, item); + critical_enter(); + do { + uma_cache_bucket_t bucket; -#ifdef UMA_ZALLOC_DEBUG - KASSERT((zone->uz_flags & UMA_ZONE_SMR) != 0, - ("uma_zfree_smr: called with non-SMR zone.")); - KASSERT(item != NULL, ("uma_zfree_smr: Called with NULL pointer.")); - SMR_ASSERT_NOT_ENTERED(zone->uz_smr); - if (uma_zfree_debug(zone, item, NULL) == EJUSTRETURN) - return; + cache = &zone->uz_cpu[curcpu]; + /* + * Try to free into the allocbucket first to give LIFO + * ordering for cache-hot datastructures. Spill over + * into the freebucket if necessary. Alloc will swap + * them if one runs dry. + */ + bucket = &cache->uc_allocbucket; +#ifdef NUMA + if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0 && + PCPU_GET(domain) != itemdomain) { + bucket = &cache->uc_crossbucket; + } else #endif - cache = &zone->uz_cpu[curcpu]; + if (bucket->ucb_cnt == bucket->ucb_entries && + cache->uc_freebucket.ucb_cnt < + cache->uc_freebucket.ucb_entries) + cache_bucket_swap(&cache->uc_freebucket, + &cache->uc_allocbucket); + if (__predict_true(bucket->ucb_cnt < bucket->ucb_entries)) { + cache_bucket_push(cache, bucket, item); + critical_exit(); + return (true); + } + } while (cache_free(zone, cache, udata, itemdomain)); + critical_exit(); + + return (false); +} + +static __always_inline bool +cache_free_smr(uma_zone_t zone, void *item, void *udata) +{ + uma_cache_t cache; + int itemdomain; +#ifdef NUMA + int uz_flags; +#endif + itemdomain = 0; #ifdef NUMA - uz_flags = cache_uz_flags(cache); + uz_flags = cache_uz_flags(&zone->uz_cpu[curcpu]); if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0) itemdomain = item_domain(item); #endif critical_enter(); do { + uma_cache_bucket_t bucket; + cache = &zone->uz_cpu[curcpu]; /* SMR Zones must free to the free bucket. */ bucket = &cache->uc_freebucket; @@ -4504,11 +4549,33 @@ uma_zfree_smr(uma_zone_t zone, void *item) if (__predict_true(bucket->ucb_cnt < bucket->ucb_entries)) { cache_bucket_push(cache, bucket, item); critical_exit(); - return; + return (true); } - } while (cache_free(zone, cache, NULL, itemdomain)); + } while (cache_free(zone, cache, udata, itemdomain)); critical_exit(); + return (false); +} + +/* See uma.h */ +void +uma_zfree_smr(uma_zone_t zone, void *item) +{ + CTR3(KTR_UMA, "uma_zfree_smr zone %s(%p) item %p", + zone->uz_name, zone, item); + +#ifdef UMA_ZALLOC_DEBUG + KASSERT((zone->uz_flags & UMA_ZONE_SMR) != 0, + ("uma_zfree_smr: called with non-SMR zone.")); + KASSERT(item != NULL, ("uma_zfree_smr: Called with NULL pointer.")); + SMR_ASSERT_NOT_ENTERED(zone->uz_smr); + if (uma_zfree_debug(zone, item, NULL) == EJUSTRETURN) + return; +#endif + + if (cache_free_smr(zone, item, NULL)) + return; + /* * If nothing else caught this, we'll just do an internal free. */ @@ -4520,8 +4587,7 @@ void uma_zfree_arg(uma_zone_t zone, void *item, void *udata) { uma_cache_t cache; - uma_cache_bucket_t bucket; - int itemdomain, uz_flags; + int uz_flags; /* Enable entropy collection for RANDOM_ENABLE_UMA kernel option */ random_harvest_fast_uma(&zone, sizeof(zone), RANDOM_UMA); @@ -4554,55 +4620,12 @@ uma_zfree_arg(uma_zone_t zone, void *item, void *udata) * The race here is acceptable. If we miss it we'll just have to wait * a little longer for the limits to be reset. */ - if (__predict_false(uz_flags & UMA_ZFLAG_LIMIT)) { - if (atomic_load_32(&zone->uz_sleepers) > 0) - goto zfree_item; - } + if (__predict_false(uz_flags & UMA_ZFLAG_LIMIT) && + atomic_load_32(&zone->uz_sleepers) > 0) + goto zfree_item; - /* - * If possible, free to the per-CPU cache. There are two - * requirements for safe access to the per-CPU cache: (1) the thread - * accessing the cache must not be preempted or yield during access, - * and (2) the thread must not migrate CPUs without switching which - * cache it accesses. We rely on a critical section to prevent - * preemption and migration. We release the critical section in - * order to acquire the zone mutex if we are unable to free to the - * current cache; when we re-acquire the critical section, we must - * detect and handle migration if it has occurred. - */ - itemdomain = 0; -#ifdef NUMA - if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0) - itemdomain = item_domain(item); -#endif - critical_enter(); - do { - cache = &zone->uz_cpu[curcpu]; - /* - * Try to free into the allocbucket first to give LIFO - * ordering for cache-hot datastructures. Spill over - * into the freebucket if necessary. Alloc will swap - * them if one runs dry. - */ - bucket = &cache->uc_allocbucket; -#ifdef NUMA - if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0 && - PCPU_GET(domain) != itemdomain) { - bucket = &cache->uc_crossbucket; - } else -#endif - if (bucket->ucb_cnt == bucket->ucb_entries && - cache->uc_freebucket.ucb_cnt < - cache->uc_freebucket.ucb_entries) - cache_bucket_swap(&cache->uc_freebucket, - &cache->uc_allocbucket); - if (__predict_true(bucket->ucb_cnt < bucket->ucb_entries)) { - cache_bucket_push(cache, bucket, item); - critical_exit(); - return; - } - } while (cache_free(zone, cache, udata, itemdomain)); - critical_exit(); + if (cache_free_item(zone, uz_flags, item, udata)) + return; /* * If nothing else caught this, we'll just do an internal free.