From nobody Tue Apr 22 09:07:08 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZhbwF2Rgjz5tpVv; Tue, 22 Apr 2025 09:07:09 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZhbwF07gVz4Nlk; Tue, 22 Apr 2025 09:07:09 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745312829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=OP70fTvMMQVmJv4KJ5tgPUyR1C3NMOf0wMbI/luVSxk=; b=XsUrrL5GNAJV6YnxaqvJkF568nxVwuXX0hdFwCQdQzxhCn41hp/a2iDzqwLCSC8+SXE8Wm 9jKaPACu9H3werYQYGDFECbWK66kDxOmpwfS6is6ZGfme6PU5wAzNO+QMelxufbT4cq6kr 1g78Tl//tAtkLc5JiukMwq4gDS4pomFa+3EjoURX5tpdzu1NJ1/IAoHEj1KOqw0ysqDlMS wZTVtxxsLLdoM1wS7pm611/YHhztO6B3e2ATTTw8Bg3vvmVuHSzprCuLGf0JR/P6xDe3jH rOcFvzt4ObNMja0dUBmz8iU0jLClusOZ4ksfB7gi27klSVqvfMxR9DQ2Cv53Rg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1745312829; a=rsa-sha256; cv=none; b=NZ7/M2ZEpXQHOT+nZNOFvNYQ+6i00yrahgoGTeGKD4qODr0ZOoaAG19WLp6ajOIcJeHqnD LynLcklfPqi11zniScsn5p585xKTYciyqRmwpQ9rqyFCMQfmnvy2ZUCK/ivDRrf01X7Hjk HsZRYdF4AnNqRcaEWdnfIea9WPqgnbMy1ttyx8v2HUfa0PjUH1xMh3qmJEzD3/hMJmJiCc +aYSFu6w6ESMHSyaHkPfmxtZR/E7RbMQe4SCzE57dCwqjKirPecM8WVkz5Xsq6hxtW1T9p AjnL6rYIImqmICrCmuvxonZ+Jtbld5KVuMwHavfdzYgdG14CPX/zBn6/dVERtw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745312829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=OP70fTvMMQVmJv4KJ5tgPUyR1C3NMOf0wMbI/luVSxk=; b=dIkp2GCFXxMlc5SarOOwCkww4EvbjYij4ayK5Ft0Y+E872F3UAackvjivZ4OFRVRKVuL8T INdMKKufOTa5tbBMRATY1hg0Zjk+atT6QTm7Ijky3/CNWsMWqnh9t5L8zq3UYu1cmLPu76 J/Y70abpUCC2m8AVahzdtwGf10CsF5SWlXnNVkDCjucqeRXHCwhHnOjKFqzoKd9IYcBBYe 3jCx9eXr7+MgQBjAXRhTGun2QfghtDulHEkSTUg+m3LSir1PdUhh0173im1rO68ZMjCJCE YTjBPm0ie2vkqRdDGoVPhUXfF9U8AulqoEoby5BCGQJaN9iOAA0Z+w/8SJ0JZQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZhbwD6rgGz1RVC; Tue, 22 Apr 2025 09:07:08 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 53M978fd065530; Tue, 22 Apr 2025 09:07:08 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 53M978VA065527; Tue, 22 Apr 2025 09:07:08 GMT (envelope-from git) Date: Tue, 22 Apr 2025 09:07:08 GMT Message-Id: <202504220907.53M978VA065527@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Ruslan Bukin Subject: git: 81e2d24bc6ea - main - riscv: support for Cache-Block Operations (CBO). List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: br X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 81e2d24bc6ea3edaa0338ea6020c2eb9f93de0ed Auto-Submitted: auto-generated The branch main has been updated by br: URL: https://cgit.FreeBSD.org/src/commit/?id=81e2d24bc6ea3edaa0338ea6020c2eb9f93de0ed commit 81e2d24bc6ea3edaa0338ea6020c2eb9f93de0ed Author: Ruslan Bukin AuthorDate: 2025-04-22 08:14:02 +0000 Commit: Ruslan Bukin CommitDate: 2025-04-22 09:03:45 +0000 riscv: support for Cache-Block Operations (CBO). CBO represents a subset of Cache-Management Operations (CMO) spec. While the CMO spec encompasses all operations on caches, the CBO subset operates on cache blocks only. Detect Zicbom, Zicboz and Zicbop extensions and provide cache invalidation handlers based on Zicbom instructions. Sponsored by: UKRI Differential Revision: https://reviews.freebsd.org/D49852 --- sys/conf/files.riscv | 1 + sys/riscv/include/cbo.h | 33 ++++++++++++++ sys/riscv/riscv/cbo.c | 104 +++++++++++++++++++++++++++++++++++++++++++++ sys/riscv/riscv/identcpu.c | 56 +++++++++++++++++++++++- 4 files changed, 192 insertions(+), 2 deletions(-) diff --git a/sys/conf/files.riscv b/sys/conf/files.riscv index 0d782239c89c..e77a15ce8dae 100644 --- a/sys/conf/files.riscv +++ b/sys/conf/files.riscv @@ -46,6 +46,7 @@ riscv/riscv/bus_space_asm.S standard riscv/riscv/busdma_bounce.c standard riscv/riscv/busdma_machdep.c standard riscv/riscv/cache.c standard +riscv/riscv/cbo.c standard riscv/riscv/clock.c standard riscv/riscv/copyinout.S standard riscv/riscv/cpufunc_asm.S standard diff --git a/sys/riscv/include/cbo.h b/sys/riscv/include/cbo.h new file mode 100644 index 000000000000..cdf10069ba66 --- /dev/null +++ b/sys/riscv/include/cbo.h @@ -0,0 +1,33 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2025 Ruslan Bukin + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#ifndef _RISCV_CBO_H_ +#define _RISCV_CBO_H_ + +void cbo_zicbom_setup_cache(int cbom_block_size); + +#endif /* _RISCV_CBO_H_ */ diff --git a/sys/riscv/riscv/cbo.c b/sys/riscv/riscv/cbo.c new file mode 100644 index 000000000000..9b8891c514af --- /dev/null +++ b/sys/riscv/riscv/cbo.c @@ -0,0 +1,104 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2025 Ruslan Bukin + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Cache Block Operations. */ + +#include +#include + +#include + +static void +cbo_zicbom_cpu_dcache_wbinv_range(vm_offset_t va, vm_size_t len) +{ + vm_offset_t addr; + + /* + * A flush operation atomically performs a clean operation followed by + * an invalidate operation. + */ + + va &= ~(dcache_line_size - 1); + for (addr = va; addr < va + len; addr += dcache_line_size) + __asm __volatile(".option push; .option arch, +zicbom\n" + "cbo.flush (%0); .option pop\n" :: "r"(addr)); +} + +static void +cbo_zicbom_cpu_dcache_inv_range(vm_offset_t va, vm_size_t len) +{ + vm_offset_t addr; + + /* + * An invalidate operation makes data from store operations performed by + * a set of non-coherent agents visible to the set of coherent agents at + * a point common to both sets by deallocating all copies of a cache + * block from the set of coherent caches up to that point. + */ + + va &= ~(dcache_line_size - 1); + for (addr = va; addr < va + len; addr += dcache_line_size) + __asm __volatile(".option push; .option arch, +zicbom\n" + "cbo.inval (%0); .option pop\n" :: "r"(addr)); +} + +static void +cbo_zicbom_cpu_dcache_wb_range(vm_offset_t va, vm_size_t len) +{ + vm_offset_t addr; + + /* + * A clean operation makes data from store operations performed by the + * set of coherent agents visible to a set of non-coherent agents at a + * point common to both sets by performing a write transfer of a copy of + * a cache block to that point provided a coherent agent performed a + * store operation that modified the data in the cache block since the + * previous invalidate, clean, or flush operation on the cache block. + */ + + va &= ~(dcache_line_size - 1); + for (addr = va; addr < va + len; addr += dcache_line_size) + __asm __volatile(".option push; .option arch, +zicbom\n" + "cbo.clean (%0); .option pop\n" :: "r"(addr)); +} + +void +cbo_zicbom_setup_cache(int cbom_block_size) +{ + struct riscv_cache_ops zicbom_ops; + + if (cbom_block_size <= 0 || !powerof2(cbom_block_size)) { + printf("Zicbom: could not initialise (invalid cache line %d)\n", + cbom_block_size); + return; + } + + zicbom_ops.dcache_wbinv_range = cbo_zicbom_cpu_dcache_wbinv_range; + zicbom_ops.dcache_inv_range = cbo_zicbom_cpu_dcache_inv_range; + zicbom_ops.dcache_wb_range = cbo_zicbom_cpu_dcache_wb_range; + riscv_cache_install_hooks(&zicbom_ops, cbom_block_size); +} diff --git a/sys/riscv/riscv/identcpu.c b/sys/riscv/riscv/identcpu.c index 54e008122eab..c76732b2ef46 100644 --- a/sys/riscv/riscv/identcpu.c +++ b/sys/riscv/riscv/identcpu.c @@ -53,6 +53,7 @@ #include #include #include +#include #ifdef FDT #include @@ -78,6 +79,11 @@ bool __read_frequently has_sstc; bool __read_frequently has_sscofpmf; bool has_svpbmt; +/* Z-extensions support. */ +bool has_zicbom; +bool has_zicboz; +bool has_zicbop; + struct cpu_desc { const char *cpu_mvendor_name; const char *cpu_march_name; @@ -89,6 +95,12 @@ struct cpu_desc { #define SV_SVPBMT (1 << 2) #define SV_SVINVAL (1 << 3) #define SV_SSCOFPMF (1 << 4) + u_int z_extensions; /* Multi-letter extensions. */ +#define Z_ZICBOM (1 << 0) +#define Z_ZICBOZ (1 << 1) +#define Z_ZICBOP (1 << 2) + int cbom_block_size; + int cboz_block_size; }; struct cpu_desc cpu_desc[MAXCPU]; @@ -196,11 +208,24 @@ parse_ext_x(struct cpu_desc *desc __unused, char *isa, int idx, int len) static __inline int parse_ext_z(struct cpu_desc *desc __unused, char *isa, int idx, int len) { +#define CHECK_Z_EXT(str, flag) \ + do { \ + if (strncmp(&isa[idx], (str), \ + MIN(strlen(str), len - idx)) == 0) { \ + desc->z_extensions |= flag; \ + return (idx + strlen(str)); \ + } \ + } while (0) + + /* Check for known/supported extensions. */ + CHECK_Z_EXT("zicbom", Z_ZICBOM); + CHECK_Z_EXT("zicboz", Z_ZICBOZ); + CHECK_Z_EXT("zicbop", Z_ZICBOP); + +#undef CHECK_Z_EXT /* * Proceed to the next multi-letter extension or the end of the * string. - * - * TODO: parse some of these. */ while (isa[idx] != '_' && idx < len) { idx++; @@ -321,6 +346,22 @@ parse_mmu_fdt(struct cpu_desc *desc, phandle_t node) } } +static void +parse_cbo_fdt(struct cpu_desc *desc, phandle_t node) +{ + int error; + + error = OF_getencprop(node, "riscv,cbom-block-size", + &desc->cbom_block_size, sizeof(desc->cbom_block_size)); + if (error == -1) + desc->cbom_block_size = 0; + + error = OF_getencprop(node, "riscv,cboz-block-size", + &desc->cboz_block_size, sizeof(desc->cboz_block_size)); + if (error == -1) + desc->cboz_block_size = 0; +} + static void identify_cpu_features_fdt(u_int cpu, struct cpu_desc *desc) { @@ -372,6 +413,9 @@ identify_cpu_features_fdt(u_int cpu, struct cpu_desc *desc) /* Check MMU features. */ parse_mmu_fdt(desc, node); + /* Cache-block operations (CBO). */ + parse_cbo_fdt(desc, node); + /* We are done. */ break; } @@ -422,6 +466,11 @@ update_global_capabilities(u_int cpu, struct cpu_desc *desc) UPDATE_CAP(has_sscofpmf, (desc->smode_extensions & SV_SSCOFPMF) != 0); UPDATE_CAP(has_svpbmt, (desc->smode_extensions & SV_SVPBMT) != 0); + /* Z extension support. */ + UPDATE_CAP(has_zicbom, (desc->z_extensions & Z_ZICBOM) != 0); + UPDATE_CAP(has_zicboz, (desc->z_extensions & Z_ZICBOZ) != 0); + UPDATE_CAP(has_zicbop, (desc->z_extensions & Z_ZICBOP) != 0); + #undef UPDATE_CAP } @@ -506,6 +555,9 @@ identify_cpu(u_int cpu) update_global_capabilities(cpu, desc); handle_cpu_quirks(cpu, desc); + + if (has_zicbom && cpu == 0) + cbo_zicbom_setup_cache(desc->cbom_block_size); } void