From nobody Fri Oct 09 01:34:03 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4j18XX0BJwz6Z8N2 for ; Fri, 09 Oct 2026 01:34:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4j18XW5ppTz4dvK for ; Fri, 09 Oct 2026 01:34:03 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1791509643; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Qa64VgKGxfMknIzNEa9ehJ3K/+Tk9GkRrm0b3FBY7mA=; b=sONiGN3F5Nr74kPdhWUIRfbGgXCrY3kOsJxj6CbKY7NNrgiZXwltquxTbGyeFiErjclV0Q fO5tmfMySV7WajL7KPiyQayFIejbSptOuoSjsviXqXVPNAIYdmgL7QQvjgvJrxDdZcCElo TGYRwWnMbG0ZX0E0ByUajKQGLm00onEeW7c+6DzyQL3GZTlWoYYudN5TlnW8+a+G/hpaQH lbpCKG0M+IGTuLW2953E+6o92B5UzCyr9s+mH31q99KOn5BltmZ+nva7ulOLU7j/nvO8+S JENtWp4W1MrlmXsy421moB0urhZmTz7z64kADhlsNhFrTjTS0xkwSe1tke9H3A== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1791509643; b=u5XqU8o3psZMyFrqDONXlgogZIdvga6EiT4eA3q1YS29wAtIRuyH302piQO70PArJtMyMN /b2CM10hm0pDjCkKMcpILAWmrZC8PJYh5OG7+4bL3VoIXcan/mUkhTVLIfl95IBKQABmN4 WYjgJK0RdGOpU+qcIMKb8NfdbZNZjqSiNKjM3XPLfUH154tQOg5GsgPQfYuLG+orsJmwd2 e4JefSao52tkpTZD6KIhhaNR1HZBnEhLS7nuyqK8aVF1j93NbeNG4IEcpJMDeScT3NqaMm NOw8V73LN2P5pOqzcLVK/42Bfj/4hW3u4hvEpdbV+JZ2rExdtRJ5ZKyLhADJJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1791509643; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Qa64VgKGxfMknIzNEa9ehJ3K/+Tk9GkRrm0b3FBY7mA=; b=pPSsEk9RxWVU4UBpxXvLUJc0UavsfJniHpwLNGXOlDsvWU3mHIs/dZ1OSFK/fENMKsNeQ7 vgidWWepH+NMC2XL/cJsO04qXzoCrVa4foCR5pgfhgVk+YbXXz4Vz/vMkrp9wcr/+eZKU7 JK8UFgYFISpwcDXwrZiYbIuEYrnR7Qpmc2BRbaIh2IAJeH/IRc+BfPpQmtXNyuiwbvqKQL K3HJDqqto7a3Nt5ub4neP/AzDTodZHgVEZhZ4z+iW+39bEAL5gOz4e7rjI9njhu6/AiBSE WzT0DDc8zCRyl5ArlQCZnqKCAF6+XzOYRIqhogciunw8z9FLfGOKuQy6rb2BgA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4j18XW4rdfz152y for ; Fri, 09 Oct 2026 01:34:03 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 22d11 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 09 Oct 2026 01:34:03 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: a66236edbf71 - main - arm64: avoid full icache flushes on PIPT icaches List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: a66236edbf7182fcb1407590a8021f1e40adc05d Auto-Submitted: auto-generated Date: Fri, 09 Oct 2026 01:34:03 +0000 Message-Id: <6ac8448b.22d11.273a1352@gitrepo.freebsd.org> The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=a66236edbf7182fcb1407590a8021f1e40adc05d commit a66236edbf7182fcb1407590a8021f1e40adc05d Author: Alan Cox AuthorDate: 2026-09-29 15:37:35 +0000 Commit: Alan Cox CommitDate: 2026-10-09 01:31:20 +0000 arm64: avoid full icache flushes on PIPT icaches On many arm64 processors, the kernel plays a part in maintaining instruction cache (icache) coherence as the contents of the underlying physical pages change. Previously, the only mechanism for maintaining icache coherence implemented by the machine-dependent layer was a full icache flush. However, on machines with physically indexed and physically tagged (PIPT) icaches, where aliasing cannot occur, we can instead selectively invalidate just the affected cache lines. Reviewed by: andrew Tested by: markj MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D60271 --- sys/arm64/arm64/cpufunc_asm.S | 81 ++++++++++++++++++++++++++++++++++--------- sys/arm64/arm64/identcpu.c | 16 ++++++--- sys/arm64/include/cpufunc.h | 2 ++ 3 files changed, 78 insertions(+), 21 deletions(-) diff --git a/sys/arm64/arm64/cpufunc_asm.S b/sys/arm64/arm64/cpufunc_asm.S index 1b5a61204c1b..cf539a2a891c 100644 --- a/sys/arm64/arm64/cpufunc_asm.S +++ b/sys/arm64/arm64/cpufunc_asm.S @@ -48,14 +48,31 @@ .Lpage_mask: .word PAGE_MASK +/* + * Macro to apply "insn op" to each cache line that overlaps the address + * range ["addr", "addr" + "size"), where "size" is non-zero. This macro + * expects the cache line size to be in x3 and "addr" to be aligned to that + * size. It will corrupt "addr" and "size". + */ +.macro apply_to_range insn, op, addr, size +1: + \insn \op, \addr + add \addr, \addr, x3 /* Move to the next line */ + subs \size, \size, x3 /* Reduce the size */ + b.hi 1b /* Check if we are done */ +.endm + /* * Macro to handle the cache. This takes the start address in x0, length * in x1. It will corrupt x0, x1, x2, x3, and x4. */ -.macro cache_handle_range dcop = 0, ic = 0, icop = 0 +.macro cache_handle_range dc = 1, dcop = 0, ic = 0, icop = 0 .if \ic == 0 adrp x3, dcache_line_size /* Load the D cache line size */ ldr x3, [x3, :lo12:dcache_line_size] +.elseif \dc == 0 + adrp x3, icache_line_size /* Load the I cache line size */ + ldr x3, [x3, :lo12:icache_line_size] .else adrp x3, idcache_line_size /* Load the I & D cache line size */ ldr x3, [x3, :lo12:idcache_line_size] @@ -64,22 +81,20 @@ and x2, x0, x4 /* Get the low bits of the address */ add x1, x1, x2 /* Add these to the size */ bic x0, x0, x4 /* Clear the low bit of the address */ +.if \dc != 0 .if \ic != 0 mov x2, x0 /* Save the address */ mov x4, x1 /* Save the size */ .endif -1: - dc \dcop, x0 - add x0, x0, x3 /* Move to the next line */ - subs x1, x1, x3 /* Reduce the size */ - b.hi 1b /* Check if we are done */ + apply_to_range dc, \dcop, x0, x1 dsb ish +.endif .if \ic != 0 -2: - ic \icop, x2 - add x2, x2, x3 /* Move to the next line */ - subs x4, x4, x3 /* Reduce the size */ - b.hi 2b /* Check if we are done */ +.if \dc != 0 + apply_to_range ic, \icop, x2, x4 +.else + apply_to_range ic, \icop, x0, x1 +.endif dsb ish isb .endif @@ -139,15 +154,35 @@ ENTRY(arm64_idc_aliasing_icache_sync_range) ret END(arm64_idc_aliasing_icache_sync_range) +/* + * Invalidating a PIPT icache by virtual address broadcasts an invalidation + * for each line to the other CPUs. For a range larger than 32KB, such as + * a superpage, invalidating the entire icache is likely to be cheaper. + */ +#define ICACHE_SYNC_RANGE_MAX (32 * 1024) + +/* + * void arm64_idc_pipt_icache_sync_range(void *, vm_size_t) + * + * When the CTR_EL0.IDC bit is set cleaning to PoU becomes a dsb. + * A PIPT icache can be invalidated by virtual address, but a large range + * is better handled by arm64_idc_aliasing_icache_sync_range()'s ialluis. + */ +ENTRY(arm64_idc_pipt_icache_sync_range) + cmp x1, #ICACHE_SYNC_RANGE_MAX + b.hi arm64_idc_aliasing_icache_sync_range + dsb ishst + cache_handle_range dc = 0, ic = 1, icop = ivau + ret +END(arm64_idc_pipt_icache_sync_range) + /* * void arm64_aliasing_icache_sync_range(void *, vm_size_t) + * + * Invalidating an aliasing icache by virtual address could miss lines that + * were fetched through a different mapping, so invalidate the entire icache. */ ENTRY(arm64_aliasing_icache_sync_range) - /* - * XXX Temporary solution - I-cache flush should be range based for - * PIPT cache or IALLUIS for VIVT or VIPT caches - */ -/* cache_handle_range dcop = cvau, ic = 1, icop = ivau */ cache_handle_range dcop = cvau ic ialluis dsb ish @@ -155,13 +190,25 @@ ENTRY(arm64_aliasing_icache_sync_range) ret END(arm64_aliasing_icache_sync_range) +/* + * void arm64_pipt_icache_sync_range(void *, vm_size_t) + * + * A PIPT icache can be invalidated by virtual address, but a large range + * is better handled by arm64_aliasing_icache_sync_range()'s ialluis. + */ +ENTRY(arm64_pipt_icache_sync_range) + cmp x1, #ICACHE_SYNC_RANGE_MAX + b.hi arm64_aliasing_icache_sync_range + cache_handle_range dcop = cvau, ic = 1, icop = ivau + ret +END(arm64_pipt_icache_sync_range) + /* * int arm64_icache_sync_range_checked(void *, vm_size_t) */ ENTRY(arm64_icache_sync_range_checked) adr x5, cache_maint_fault SET_FAULT_HANDLER(x5, x6) - /* XXX: See comment in arm64_icache_sync_range */ cache_handle_range dcop = cvau ic ialluis dsb ish diff --git a/sys/arm64/arm64/identcpu.c b/sys/arm64/arm64/identcpu.c index 730fd265cfb0..d5c38d9b7dd6 100644 --- a/sys/arm64/arm64/identcpu.c +++ b/sys/arm64/arm64/identcpu.c @@ -92,7 +92,7 @@ static uint64_t update_special_reg_field(uint64_t user_reg, u_int type, * aliasing cache until we know otherwise. */ void (*arm64_icache_sync_range)(void *, vm_size_t) = - &arm64_aliasing_icache_sync_range; + arm64_aliasing_icache_sync_range; static int sysctl_hw_machine(SYSCTL_HANDLER_ARGS) @@ -2828,13 +2828,21 @@ identify_cpu_sysinit(void *dummy __unused) } if (dic && idc) { - arm64_icache_sync_range = &arm64_dic_idc_icache_sync_range; + arm64_icache_sync_range = arm64_dic_idc_icache_sync_range; if (bootverbose) printf("Enabling DIC & IDC ICache sync\n"); - } else if (idc) { - arm64_icache_sync_range = &arm64_idc_aliasing_icache_sync_range; + } else if (idc && icache_aliasing) { + arm64_icache_sync_range = arm64_idc_aliasing_icache_sync_range; if (bootverbose) printf("Enabling IDC ICache sync\n"); + } else if (idc) { + arm64_icache_sync_range = arm64_idc_pipt_icache_sync_range; + if (bootverbose) + printf("Enabling IDC & PIPT ICache sync\n"); + } else if (!icache_aliasing) { + arm64_icache_sync_range = arm64_pipt_icache_sync_range; + if (bootverbose) + printf("Enabling PIPT ICache sync\n"); } } /* diff --git a/sys/arm64/include/cpufunc.h b/sys/arm64/include/cpufunc.h index 91e6ad3d8cde..f6e058b7b52b 100644 --- a/sys/arm64/include/cpufunc.h +++ b/sys/arm64/include/cpufunc.h @@ -214,7 +214,9 @@ extern void (*arm64_icache_sync_range)(void *, vm_size_t); void arm64_nullop(void); void arm64_dic_idc_icache_sync_range(void *, vm_size_t); void arm64_idc_aliasing_icache_sync_range(void *, vm_size_t); +void arm64_idc_pipt_icache_sync_range(void *, vm_size_t); void arm64_aliasing_icache_sync_range(void *, vm_size_t); +void arm64_pipt_icache_sync_range(void *, vm_size_t); int arm64_icache_sync_range_checked(void *, vm_size_t); void arm64_dcache_wbinv_range(void *, vm_size_t); void arm64_dcache_inv_range(void *, vm_size_t);