From nobody Tue Mar 01 14:40:09 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D5E7019DBB35; Tue, 1 Mar 2022 14:40:10 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K7KfK3H0nz3Fqy; Tue, 1 Mar 2022 14:40:09 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1646145609; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=kmLBIF9TXLftkpEd1y2u5ecTzmVRss3IefjoqongLQM=; b=F6Uz02J2awEh81zCKiL24tlbMQooNudBiRk4vvYH33TnE5g0sRb2lgYc/gDSa/+A7zgSgQ lAG7CAhL7iYJGwj3V86ZVGFM/kdbwzwmUka2O01ampWa7GO/Pyh715SiASDV7j5pwhS+Mq PlJbym13zaAzI0MqUrzVI2Pw7EKlynwo0tzTqvEtfR4Mfzf207NkRb/IcI1hascYkNM2Xa cImHwakOAd4lYR5QWyT2p1ZOyz6oTrlG1xKcYlUQxsEdjC9SKXgydBLFnqyi206Eztxgbx 4kVVxdYNLAvwPeYaAFKqdNpDrMqAqjZGi+3TqAPqzLMRm670/NcvsMSNgH65cQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2EE381146E; Tue, 1 Mar 2022 14:40:09 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 221Ee9eQ041804; Tue, 1 Mar 2022 14:40:09 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 221Ee9CI041802; Tue, 1 Mar 2022 14:40:09 GMT (envelope-from git) Date: Tue, 1 Mar 2022 14:40:09 GMT Message-Id: <202203011440.221Ee9CI041802@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 31218f3209ac - main - riscv: Add support for enabling SV48 mode List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 31218f3209acf0f8f0dc54e436342c8fa604d279 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1646145609; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=kmLBIF9TXLftkpEd1y2u5ecTzmVRss3IefjoqongLQM=; b=F0KyNJx7ujueZSURsBtnH0ZCLgdwKwdhClThyUqgXNh17nKqJh6ygBdQ7nfksZN92CcdC7 fPCsmdyx+RPMTgtJWohECmy1NKZOfQx0tk1wvGc01ChdA3B++D4Sq4YZSWxd8kXSYaWX/2 d/a+QVWhR8KGdK/2GXGkDYovBFEZQH+AjH8fpZQcsMBBfsav3O15s/JoaYgdFEboyvnjxZ geh4UxGz5FfDiOKTLd800+1KSP/K4KotL0HEAnHeJWZhKsaTK80TTcfmE1nMFll6MToviO mevj5C4Nv3TKF9vIVbxW3QTd3gLPiPILMKaaZReFZLx1Ye9PnT+XQfD/wqXgjw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1646145609; a=rsa-sha256; cv=none; b=jhW7+RYFxYJCPcheO1Gud5uoiAymIlAyMJdUGRtqBoewsS2K9an7ms+5XRs4ESX8WokSk+ jrqkYvzAF3mixtGYkFFl1/81K966OJhm+ynOFNdZ+B4vBrjJXvdYtnl64fNZvNz0ODnNP6 eXxPVJJQh9MMjVB1NaTND9KiRsNdJ8y1sc7nwz/+P80INzKmqzqPGqenmC/r05Am01gc0x Pl03F2gADRxGfvkPjwCcBmTWdPE5/eAQq/opeR4wP4f/p3TZREmCRjGR6NbqSrIeUpftB+ jxNvprFBR9TVY2sXsRQR4V8/ZaFf/5e4CJYWX829YqGE1lNdQymIDLHtLxuw7A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=31218f3209acf0f8f0dc54e436342c8fa604d279 commit 31218f3209acf0f8f0dc54e436342c8fa604d279 Author: Mark Johnston AuthorDate: 2022-03-01 14:07:14 +0000 Commit: Mark Johnston CommitDate: 2022-03-01 14:39:44 +0000 riscv: Add support for enabling SV48 mode This increases the size of the user map from 256GB to 128TB. The kernel map is left unchanged for now. For now SV48 mode is left disabled by default, but can be enabled with a tunable. Note that extant hardware does not implement SV48, but QEMU does. - In pmap_bootstrap(), allocate a L0 page and attempt to enable SV48 mode. If the write to SATP doesn't take, the kernel continues to run in SV39 mode. - Define VM_MAX_USER_ADDRESS to refer to the SV48 limit. In SV39 mode, the region [VM_MAX_USER_ADDRESS_SV39, VM_MAX_USER_ADDRESS_SV48] is not mappable. Reviewed by: jhb MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D34280 --- sys/riscv/include/riscvreg.h | 6 ++--- sys/riscv/include/vmparam.h | 6 ++--- sys/riscv/riscv/pmap.c | 59 +++++++++++++++++++++++++++++++++++--------- 3 files changed, 54 insertions(+), 17 deletions(-) diff --git a/sys/riscv/include/riscvreg.h b/sys/riscv/include/riscvreg.h index e7b52cae8042..91b6f9cb3ee9 100644 --- a/sys/riscv/include/riscvreg.h +++ b/sys/riscv/include/riscvreg.h @@ -150,11 +150,11 @@ #define SIP_STIP (1 << 5) #define SATP_PPN_S 0 -#define SATP_PPN_M (0xfffffffffff << SATP_PPN_S) +#define SATP_PPN_M (0xfffffffffffUL << SATP_PPN_S) #define SATP_ASID_S 44 -#define SATP_ASID_M (0xffff << SATP_ASID_S) +#define SATP_ASID_M (0xffffUL << SATP_ASID_S) #define SATP_MODE_S 60 -#define SATP_MODE_M (0xf << SATP_MODE_S) +#define SATP_MODE_M (0xfUL << SATP_MODE_S) #define SATP_MODE_SV39 (8ULL << SATP_MODE_S) #define SATP_MODE_SV48 (9ULL << SATP_MODE_S) diff --git a/sys/riscv/include/vmparam.h b/sys/riscv/include/vmparam.h index 35f82638cd70..f11f02dcb3e6 100644 --- a/sys/riscv/include/vmparam.h +++ b/sys/riscv/include/vmparam.h @@ -197,7 +197,7 @@ #define VM_MIN_USER_ADDRESS (0x0000000000000000UL) #define VM_MAX_USER_ADDRESS_SV39 (0x0000004000000000UL) #define VM_MAX_USER_ADDRESS_SV48 (0x0000800000000000UL) -#define VM_MAX_USER_ADDRESS VM_MAX_USER_ADDRESS_SV39 +#define VM_MAX_USER_ADDRESS VM_MAX_USER_ADDRESS_SV48 #define VM_MINUSER_ADDRESS (VM_MIN_USER_ADDRESS) #define VM_MAXUSER_ADDRESS (VM_MAX_USER_ADDRESS) @@ -209,10 +209,10 @@ #define KERNBASE (VM_MIN_KERNEL_ADDRESS) #define SHAREDPAGE_SV39 (VM_MAX_USER_ADDRESS_SV39 - PAGE_SIZE) #define SHAREDPAGE_SV48 (VM_MAX_USER_ADDRESS_SV48 - PAGE_SIZE) -#define SHAREDPAGE SHAREDPAGE_SV39 +#define SHAREDPAGE SHAREDPAGE_SV48 #define USRSTACK_SV39 SHAREDPAGE_SV39 #define USRSTACK_SV48 SHAREDPAGE_SV48 -#define USRSTACK USRSTACK_SV39 +#define USRSTACK USRSTACK_SV48 #define PS_STRINGS_SV39 (USRSTACK_SV39 - sizeof(struct ps_strings)) #define PS_STRINGS_SV48 (USRSTACK_SV48 - sizeof(struct ps_strings)) diff --git a/sys/riscv/riscv/pmap.c b/sys/riscv/riscv/pmap.c index 3757f6de3043..ef33ea6719b8 100644 --- a/sys/riscv/riscv/pmap.c +++ b/sys/riscv/riscv/pmap.c @@ -228,11 +228,17 @@ __FBSDID("$FreeBSD$"); #define VM_PAGE_TO_PV_LIST_LOCK(m) \ PHYS_TO_PV_LIST_LOCK(VM_PAGE_TO_PHYS(m)) +static SYSCTL_NODE(_vm, OID_AUTO, pmap, CTLFLAG_RD | CTLFLAG_MPSAFE, 0, + "VM/pmap parameters"); + /* The list of all the user pmaps */ LIST_HEAD(pmaplist, pmap); static struct pmaplist allpmaps = LIST_HEAD_INITIALIZER(); enum pmap_mode __read_frequently pmap_mode = PMAP_MODE_SV39; +SYSCTL_INT(_vm_pmap, OID_AUTO, mode, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, + &pmap_mode, 0, + "translation mode, 0 = SV39, 1 = SV48"); struct pmap kernel_pmap_store; @@ -251,9 +257,6 @@ CTASSERT((DMAP_MAX_ADDRESS & ~L1_OFFSET) == DMAP_MAX_ADDRESS); static struct rwlock_padalign pvh_global_lock; static struct mtx_padalign allpmaps_lock; -static SYSCTL_NODE(_vm, OID_AUTO, pmap, CTLFLAG_RD | CTLFLAG_MPSAFE, 0, - "VM/pmap parameters"); - static int superpages_enabled = 1; SYSCTL_INT(_vm_pmap, OID_AUTO, superpages_enabled, CTLFLAG_RDTUN, &superpages_enabled, 0, @@ -617,12 +620,13 @@ pmap_bootstrap_l3(vm_offset_t l1pt, vm_offset_t va, vm_offset_t l3_start) void pmap_bootstrap(vm_offset_t l1pt, vm_paddr_t kernstart, vm_size_t kernlen) { - u_int l1_slot, l2_slot; - vm_offset_t freemempos; - vm_offset_t dpcpu, msgbufpv; - vm_paddr_t max_pa, min_pa, pa; + uint64_t satp; + vm_offset_t dpcpu, freemempos, l0pv, msgbufpv; + vm_paddr_t l0pa, l1pa, max_pa, min_pa, pa; + pd_entry_t *l0p; pt_entry_t *l2p; - int i; + u_int l1_slot, l2_slot; + int i, mode; printf("pmap_bootstrap %lx %lx %lx\n", l1pt, kernstart, kernlen); @@ -696,6 +700,33 @@ pmap_bootstrap(vm_offset_t l1pt, vm_paddr_t kernstart, vm_size_t kernlen) freemempos += (np * PAGE_SIZE); \ memset((char *)(var), 0, ((np) * PAGE_SIZE)); + mode = 0; + TUNABLE_INT_FETCH("vm.pmap.mode", &mode); + if (mode == PMAP_MODE_SV48) { + /* + * Enable SV48 mode: allocate an L0 page and set SV48 mode in + * SATP. If the implementation does not provide SV48 mode, + * the mode read back from the (WARL) SATP register will be + * unchanged, and we continue in SV39 mode. + */ + alloc_pages(l0pv, 1); + l0p = (void *)l0pv; + l1pa = pmap_early_vtophys(l1pt, l1pt); + l0p[pmap_l0_index(KERNBASE)] = PTE_V | PTE_A | PTE_D | + ((l1pa >> PAGE_SHIFT) << PTE_PPN0_S); + + l0pa = pmap_early_vtophys(l1pt, l0pv); + csr_write(satp, (l0pa >> PAGE_SHIFT) | SATP_MODE_SV48); + satp = csr_read(satp); + if ((satp & SATP_MODE_M) == SATP_MODE_SV48) { + pmap_mode = PMAP_MODE_SV48; + kernel_pmap_store.pm_top = l0p; + } else { + /* Mode didn't change, give the page back. */ + freemempos -= PAGE_SIZE; + } + } + /* Allocate dynamic per-cpu area. */ alloc_pages(dpcpu, DPCPU_SIZE / PAGE_SIZE); dpcpu_init((void *)dpcpu, 0); @@ -1269,14 +1300,20 @@ pmap_unuse_pt(pmap_t pmap, vm_offset_t va, pd_entry_t ptepde, return (pmap_unwire_ptp(pmap, va, mpte, free)); } +static uint64_t +pmap_satp_mode(void) +{ + return (pmap_mode == PMAP_MODE_SV39 ? SATP_MODE_SV39 : SATP_MODE_SV48); +} + void pmap_pinit0(pmap_t pmap) { - PMAP_LOCK_INIT(pmap); bzero(&pmap->pm_stats, sizeof(pmap->pm_stats)); pmap->pm_top = kernel_pmap->pm_top; - pmap->pm_satp = SATP_MODE_SV39 | (vtophys(pmap->pm_top) >> PAGE_SHIFT); + pmap->pm_satp = pmap_satp_mode() | + (vtophys(pmap->pm_top) >> PAGE_SHIFT); CPU_ZERO(&pmap->pm_active); pmap_activate_boot(pmap); } @@ -1293,7 +1330,7 @@ pmap_pinit(pmap_t pmap) topphys = VM_PAGE_TO_PHYS(mtop); pmap->pm_top = (pd_entry_t *)PHYS_TO_DMAP(topphys); - pmap->pm_satp = SATP_MODE_SV39 | (topphys >> PAGE_SHIFT); + pmap->pm_satp = pmap_satp_mode() | (topphys >> PAGE_SHIFT); bzero(&pmap->pm_stats, sizeof(pmap->pm_stats));