From nobody Mon Nov 01 13:52:16 2021 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 20822183E408; Mon, 1 Nov 2021 13:52:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HjZGT4WbBz4pXp; Mon, 1 Nov 2021 13:52:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1416B12B20; Mon, 1 Nov 2021 13:52:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1A1DqGiS066847; Mon, 1 Nov 2021 13:52:16 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1A1DqGXN066846; Mon, 1 Nov 2021 13:52:16 GMT (envelope-from git) Date: Mon, 1 Nov 2021 13:52:16 GMT Message-Id: <202111011352.1A1DqGXN066846@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: fab343a7168a - main - uma: Improve M_USE_RESERVE handling in keg_fetch_slab() List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: fab343a7168a2f033073bb5f65b5af17d9092c6f Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=fab343a7168a2f033073bb5f65b5af17d9092c6f commit fab343a7168a2f033073bb5f65b5af17d9092c6f Author: Mark Johnston AuthorDate: 2021-11-01 13:27:35 +0000 Commit: Mark Johnston CommitDate: 2021-11-01 13:51:18 +0000 uma: Improve M_USE_RESERVE handling in keg_fetch_slab() M_USE_RESERVE is used in a couple of places in the VM to avoid unbounded recursion when the direct map is not available, as is the case on 32-bit platforms or when certain kernel sanitizers (KASAN and KMSAN) are enabled. For example, to allocate KVA, the kernel might allocate a kernel map entry, which might require a new slab, which requires KVA. For these zones, we use uma_prealloc() to populate a reserve of items, and then in certain serialized contexts M_USE_RESERVE can be used to guarantee a successful allocation. uma_prealloc() allocates the requested number of items, distributing them evenly among NUMA domains. Thus, in a first-touch zone, to satisfy an M_USE_RESERVE allocation we might have to check the slab lists of other domains than the current one to provide the semantics expected by consumers. So, try harder to find an item if M_USE_RESERVE is specified and the keg doesn't have anything for current (first-touch) domain. Specifically, fall back to a round-robin slab allocation. This change fixes boot-time panics on NUMA systems with KASAN or KMSAN enabled.[1] Alternately we could have uma_prealloc() allocate the requested number of items for each domain, but for some existing consumers this would be quite wasteful. In general I think keg_fetch_slab() should try harder to find free slabs in other domains before trying to allocate fresh ones, but let's limit this to M_USE_RESERVE for now. Also fix a separate problem that I noticed: in a non-round-robin slab allocation with M_WAITOK, rather than sleeping after a failed slab allocation we simply try again. Call vm_wait_domain() before retrying. Reported by: mjg, tuexen [1] Reviewed by: alc MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D32515 --- sys/vm/uma_core.c | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/sys/vm/uma_core.c b/sys/vm/uma_core.c index 35ed473da5ca..de9605a28bb6 100644 --- a/sys/vm/uma_core.c +++ b/sys/vm/uma_core.c @@ -3914,6 +3914,9 @@ keg_fetch_slab(uma_keg_t keg, uma_zone_t zone, int rdomain, const int flags) int aflags, domain; bool rr; + KASSERT((flags & (M_WAITOK | M_NOVM)) != (M_WAITOK | M_NOVM), + ("%s: invalid flags %#x", __func__, flags)); + restart: /* * Use the keg's policy if upper layers haven't already specified a @@ -3939,17 +3942,29 @@ restart: return (slab); /* - * M_NOVM means don't ask at all! + * M_NOVM is used to break the recursion that can otherwise + * occur if low-level memory management routines use UMA. */ - if (flags & M_NOVM) - break; + if ((flags & M_NOVM) == 0) { + slab = keg_alloc_slab(keg, zone, domain, flags, aflags); + if (slab != NULL) + return (slab); + } - slab = keg_alloc_slab(keg, zone, domain, flags, aflags); - if (slab != NULL) - return (slab); - if (!rr && (flags & M_WAITOK) == 0) - break; - if (rr && vm_domainset_iter_policy(&di, &domain) != 0) { + if (!rr) { + if ((flags & M_USE_RESERVE) != 0) { + /* + * Drain reserves from other domains before + * giving up or sleeping. It may be useful to + * support per-domain reserves eventually. + */ + rdomain = UMA_ANYDOMAIN; + goto restart; + } + if ((flags & M_WAITOK) == 0) + break; + vm_wait_domain(domain); + } else if (vm_domainset_iter_policy(&di, &domain) != 0) { if ((flags & M_WAITOK) != 0) { vm_wait_doms(&keg->uk_dr.dr_policy->ds_mask, 0); goto restart;