From nobody Sat Aug 29 06:34:57 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hX58f1MCyz6qSmx for ; Sat, 29 Aug 2026 06:34:58 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hX58d5ykXz3M3d for ; Sat, 29 Aug 2026 06:34:57 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787985297; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=vaC8XP3LihStWmlGMjdSkWSikWjh/LPTxc2xFEfhtZ8=; b=F7lYHeEEGLAjReSgzYMBge3oNBrXbehGcf28fonqJe2AlNMHezDsRQ8vv/RFBVUYJwO0Qm uzmqPld6PYgefRbFsZc/R+78PCmT3wFIigloM2PoenC6QdCW1zxujwSJ9QUGACagTQmbpV 2UIXIPnVtbaJ8JUlz3vUSlOg/Hq7BFPRlaG895iQ2q889B+JIUBhpWArs0cYrtlPKxpbjM CZJDBqZUuZEEGQSjJNo/4QlypLesfFM4rnAKIMLEGJJH/ZX+k7Cj7sVYoublNdWx7Sw9+5 Rc7QOGjSqCH6QiBcEAVLZy0jiVZJK9nCT74yTNigSoWALbdriVGf/63zaMo45A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1787985297; a=rsa-sha256; cv=none; b=W+q7fL2y+oLTOl0gYvFRJc0Zt8TtVzO4TPoMFSz6wLGovDZkxNeEPNjlperPljduDA45cL TmehT13IdM9TzggUanCetGbVo7ioKo7WIN+cty0lp56eAMTPQIpJJGIerGdOTpxudfvZwt ltuDCSLeP6+VbhRravqQe7/HmY2rs4x+3wcR076vB5fYs5zvETBQp+Jbjn876feGn3dNwh v7ORuozvSdeHrz/5XfssoMCu7q1DQBr/9Zbq+jDKTyKBQSPdAvw1cUhGsM/qMXZMWNO4bm U8ycgUQTdzk3NetAvA3nNXHkJVv0X9qkeJlsFzZTOUJI7AHy8/qkvAOmoZNJQQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787985297; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=vaC8XP3LihStWmlGMjdSkWSikWjh/LPTxc2xFEfhtZ8=; b=XHF2TiKB19Hudf3EX9jCX0PVbc4Ty/bZdmPjJqzhK+d7yYZRwyJAipAeevNe7wwxHawjgb 4Uq89d2R7Q6WmfVaC6flhgcOGJh30R0YJdYdCSxpSIb+prXms9fHRf+syWd3jsQvGlPHPV ZYDd6ittg3qElG7CY8DcLpwKvAPzesdEYO3W3bmWSJIEyeLe5jvt63fdPvZGcWL31xBp0V FWQD8m5UwbeA39fpn/LR64FTS4L1tFMWqOMwWzb4MH7ialXRbg3iRiM2bUptP7y77cpvFU RtQkzmBS9TR+UbBzJsUzoM/3AoJDkhkfE8rm5EapsDaCoasUiXjkuhMDRo7HRg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hX58d4nD9zTXy for ; Sat, 29 Aug 2026 06:34:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 33e2e by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Sat, 29 Aug 2026 06:34:57 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: b2e6b6545ee6 - main - arm64 pmap: optimize TLB management by pmap_update_entry() List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b2e6b6545ee6377a1c8bc2b0a5bcc4cb605a8283 Auto-Submitted: auto-generated Date: Sat, 29 Aug 2026 06:34:57 +0000 Message-Id: <6a927d91.33e2e.85b6315@gitrepo.freebsd.org> The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=b2e6b6545ee6377a1c8bc2b0a5bcc4cb605a8283 commit b2e6b6545ee6377a1c8bc2b0a5bcc4cb605a8283 Author: Alan Cox AuthorDate: 2026-08-15 17:43:43 +0000 Commit: Alan Cox CommitDate: 2026-08-29 06:33:22 +0000 arm64 pmap: optimize TLB management by pmap_update_entry() To date, pmap_update_entry() has unconditionally passed false as final_only to pmap_s1_invalidate_range(). Passing false means that we invalidate the intermediate "page walk cache" entries in the TLB as well as the leaf that is being replaced. However, invalidating intermediate entries is only necessary when doing a superpage promotion that replaces a pointer to a page table page by a large page mapping. Reviewed by: andrew, kib, markj MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D58917 --- sys/arm64/arm64/pmap.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index 30b7f5b6d7d9..8f68be2fb272 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -592,7 +592,7 @@ static void _pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t m, struct spglist *free); static int pmap_unuse_pt(pmap_t, vm_offset_t, pd_entry_t, struct spglist *); static void pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_entry_t newpte, - vm_offset_t va, vm_size_t size); + vm_offset_t va, vm_size_t size, bool final_only); static __inline vm_page_t pmap_remove_pt_page(pmap_t pmap, vm_offset_t va); static uma_zone_t pmap_bti_ranges_zone; @@ -2646,7 +2646,7 @@ pmap_kenter(vm_offset_t sva, vm_size_t size, vm_paddr_t pa, int mode) */ pmap_update_entry(kernel_pmap, pde, PHYS_TO_PTE(pa) | attr | L2_BLOCK, va, - PAGE_SIZE); + PAGE_SIZE, false); } PMAP_UNLOCK(kernel_pmap); if (error == 0) { @@ -5081,11 +5081,15 @@ pmap_remove_pt_page(pmap_t pmap, vm_offset_t va) /* * Performs a break-before-make update of a pmap entry. This is needed when * either promoting or demoting pages to ensure the TLB doesn't get into an - * inconsistent state. + * inconsistent state. The caller must pass false for "final_only" when + * promoting, because the TLB might be caching an intermediate entry that + * references the L{1,2}_TABLE that is being replaced. In contrast, when + * demoting or the PTE's type isn't changing, no cached intermediate entry + * needs to change, so the caller should pass true as an optimization. */ -static void +static __always_inline void pmap_update_entry(pmap_t pmap, pd_entry_t *ptep, pd_entry_t newpte, - vm_offset_t va, vm_size_t size) + vm_offset_t va, vm_size_t size, bool final_only) { register_t intr; @@ -5108,11 +5112,10 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *ptep, pd_entry_t newpte, pmap_clear_bits(ptep, ATTR_DESCR_VALID); /* - * When promoting, the L{1,2}_TABLE entry that is being replaced might - * be cached, so we invalidate intermediate entries as well as final - * entries. + * We always inline pmap_update_entry() so that constant propagation + * and dead code elimination will specialize the following code. */ - pmap_s1_invalidate_range(pmap, va, va + size, false); + pmap_s1_invalidate_range(pmap, va, va + size, final_only); /* Create the new mapping */ pmap_store(ptep, newpte); @@ -5346,7 +5349,8 @@ setl3: if ((newl2 & ATTR_SW_MANAGED) != 0) pmap_pv_promote_l2(pmap, va, PTE_TO_PHYS(newl2), lockp); - pmap_update_entry(pmap, l2, newl2 | L2_BLOCK, va & ~L2_OFFSET, L2_SIZE); + pmap_update_entry(pmap, l2, newl2 | L2_BLOCK, va & ~L2_OFFSET, L2_SIZE, + false); counter_u64_add(pmap_l2_promotions, 1); CTR2(KTR_PMAP, "pmap_promote_l2: success for va %#lx in pmap %p", va, @@ -8735,7 +8739,7 @@ pmap_change_props_locked(void *addr, vm_size_t size, vm_prot_t prot, * performed. */ pmap_update_entry(kernel_pmap, ptep, pte, tmpva, - PAGE_SIZE); + PAGE_SIZE, true); break; } @@ -8847,7 +8851,7 @@ pmap_demote_l1(pmap_t pmap, pt_entry_t *l1, vm_offset_t va) l1 = (pt_entry_t *)(tmpl1 + ((vm_offset_t)l1 & PAGE_MASK)); } - pmap_update_entry(pmap, l1, l2phys | L1_TABLE, va, PAGE_SIZE); + pmap_update_entry(pmap, l1, l2phys | L1_TABLE, va, PAGE_SIZE, true); counter_u64_add(pmap_l1_demotions, 1); fail: @@ -9057,7 +9061,7 @@ pmap_demote_l2_locked(pmap_t pmap, pt_entry_t *l2, vm_offset_t va, * Pass PAGE_SIZE so that a single TLB invalidation is performed on * the 2MB page mapping. */ - pmap_update_entry(pmap, l2, l3phys | L2_TABLE, va, PAGE_SIZE); + pmap_update_entry(pmap, l2, l3phys | L2_TABLE, va, PAGE_SIZE, true); /* * Demote the PV entry.