From nobody Sat Dec 02 21:55:56 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SjNzK1Pw8z52nnv; Sat, 2 Dec 2023 21:55:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SjNzK0zL2z4plb; Sat, 2 Dec 2023 21:55:57 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701554157; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=SGRStjCWjuE+dVFmKPgA8DPgcPQ4BJQtTB0JtmUfu/8=; b=rFn3seHkIe1GwLEMgGo/kOMhwUlTDrQ+lRN7L3+9qHOwdEbHFHYTYFWmZTDofTDsN6OcEn W3nhJ66MX4RoFGPl6drBOQeweiD/zS6lmJNoHzvVDx01WzCqeu8MI73EQPQtmDSErbWAS9 BLkLEK2wIQexCtLgZg1JvGJ58V+9ZQWrjHtyJoq6f7Y7umgFCTKl+36aVrCjUw+oRrbN49 qN20og9JJxx33+USJqi2LjJm/WotmBwlRX1NAYs7Y8D7+ZiTGmBC4w/pxC8n2EboxrpeDM SwQiO8bmZoso5d35G4fe0HmlRCOI3o8+ysvrKWe1T0xpgwxg7p37ODNrCmM5Gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701554157; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=SGRStjCWjuE+dVFmKPgA8DPgcPQ4BJQtTB0JtmUfu/8=; b=xFhpR6JObnPNlksCuRYBV1wG0jKiNFuVrxZI3T9DsG159xhL36RuNS/5/QFWQeArWAg3SX 4oLklXsRl9OtpntuZHrVE8DDxD0CoJJ56we5dGAT+v1KhOfOKDQRbG1flyrh+d13t3Ar5p sk+uFK1ip4FAXni7ym/r43okZIi69Huox/lVV5W5vlHVwDV3jJfktUoFeHD6zttju6e6ln YluG3mB7K6vfehcIPW2pIPyZAjAxaom07QPEmYyXWCWeL+povE0p8Nax4i7EgVln5evChR HqOA33M9H5lZrPsggTbI8+/UMrR6zq7wuU/ipMVgajm3yRUbHA8ChQ/HXU17Vw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1701554157; a=rsa-sha256; cv=none; b=WJg2XUXoeCp6NzMmq7HqipRJb2OC7Pb7cKYgJLDBXe4ng31H0NhCowsP8uFTg3KrpdE7Xv pqUDK5gNo6rflDSrkxtUD9qluzXcja+papuUsq8aw5z+boMELODgTex0f1IRSGjNxChekN FJu36GaZvmRCyZh/7pGeN59+RR1odQ0HtE0tVJcAY5mMrVGg9asRe7pnm6dp0iuyaltw/8 szJ5K/WNyvnejZmr1IwoGBzJvNEUr9nhdYzzmSoJ+aoqZfMifc+AFpga6EjLWwC9qDpxXl dmEwh1C2rnYeEQHB6thqdfYPHQxNAFeT/SnmJWiZ2R5E7h0F8SpxZD/GBM2fcw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SjNzK02sJzZ2m; Sat, 2 Dec 2023 21:55:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3B2Ltu6i034670; Sat, 2 Dec 2023 21:55:56 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3B2LtukL034667; Sat, 2 Dec 2023 21:55:56 GMT (envelope-from git) Date: Sat, 2 Dec 2023 21:55:56 GMT Message-Id: <202312022155.3B2LtukL034667@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 8c88d17fa5c4 - stable/14 - riscv: Port improvements from arm64/amd64 pmaps, part 1 List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 8c88d17fa5c49a835966091acc9c41d19b1f3692 Auto-Submitted: auto-generated The branch stable/14 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=8c88d17fa5c49a835966091acc9c41d19b1f3692 commit 8c88d17fa5c49a835966091acc9c41d19b1f3692 Author: Mark Johnston AuthorDate: 2023-11-02 18:33:37 +0000 Commit: Mark Johnston CommitDate: 2023-12-02 21:54:32 +0000 riscv: Port improvements from arm64/amd64 pmaps, part 1 - When promoting, do not require that all PTEs all have PTE_A set. Instead, record whether they did and store this information in the PTP's valid bits. - Synchronize some comments in pmap_promote_l2(). - Make pmap_promote_l2() scan starting from the end of the 2MB range instead of the beginning. See the commit log for 9d1b7fa31f510 for justification of this, which I believe applies here as well. Reviewed by: kib MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D42288 (cherry picked from commit 7703ac2e983bcd44ba93878f52eb30355cbc4821) --- sys/riscv/include/pte.h | 2 +- sys/riscv/riscv/pmap.c | 66 ++++++++++++++++++++++++++++++++++++++----------- 2 files changed, 52 insertions(+), 16 deletions(-) diff --git a/sys/riscv/include/pte.h b/sys/riscv/include/pte.h index 2cede135ebae..da7bd051e122 100644 --- a/sys/riscv/include/pte.h +++ b/sys/riscv/include/pte.h @@ -80,7 +80,7 @@ typedef uint64_t pn_t; /* page number */ #define PTE_RWX (PTE_R | PTE_W | PTE_X) #define PTE_RX (PTE_R | PTE_X) #define PTE_KERN (PTE_V | PTE_R | PTE_W | PTE_A | PTE_D) -#define PTE_PROMOTE (PTE_V | PTE_RWX | PTE_D | PTE_A | PTE_G | PTE_U | \ +#define PTE_PROMOTE (PTE_V | PTE_RWX | PTE_D | PTE_G | PTE_U | \ PTE_SW_MANAGED | PTE_SW_WIRED) /* Bits 63 - 54 are reserved for future use. */ diff --git a/sys/riscv/riscv/pmap.c b/sys/riscv/riscv/pmap.c index 1d1d0fc839ca..716f00b36005 100644 --- a/sys/riscv/riscv/pmap.c +++ b/sys/riscv/riscv/pmap.c @@ -1188,14 +1188,26 @@ pmap_add_delayed_free_list(vm_page_t m, struct spglist *free, * for mapping a distinct range of virtual addresses. The pmap's collection is * ordered by this virtual address range. * - * If "promoted" is false, then the page table page "ml3" must be zero filled. + * If "promoted" is false, then the page table page "mpte" must be zero filled; + * "mpte"'s valid field will be set to 0. + * + * If "promoted" is true and "all_l3e_PTE_A_set" is false, then "mpte" must + * contain valid mappings with identical attributes except for PTE_A; + * "mpte"'s valid field will be set to 1. + * + * If "promoted" and "all_l3e_PTE_A_set" are both true, then "mpte" must contain + * valid mappings with identical attributes including PTE_A; "mpte"'s valid + * field will be set to VM_PAGE_BITS_ALL. */ static __inline int -pmap_insert_pt_page(pmap_t pmap, vm_page_t ml3, bool promoted) +pmap_insert_pt_page(pmap_t pmap, vm_page_t ml3, bool promoted, + bool all_l3e_PTE_A_set) { PMAP_LOCK_ASSERT(pmap, MA_OWNED); - ml3->valid = promoted ? VM_PAGE_BITS_ALL : 0; + KASSERT(promoted || !all_l3e_PTE_A_set, + ("a zero-filled PTP can't have PTE_A set in every PTE")); + ml3->valid = promoted ? (all_l3e_PTE_A_set ? VM_PAGE_BITS_ALL : 1) : 0; return (vm_radix_insert(&pmap->pm_root, ml3)); } @@ -2163,7 +2175,7 @@ pmap_remove_kernel_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_t va) * If this page table page was unmapped by a promotion, then it * contains valid mappings. Zero it to invalidate those mappings. */ - if (ml3->valid != 0) + if (vm_page_any_valid(ml3)) pagezero((void *)PHYS_TO_DMAP(ml3pa)); /* @@ -2224,7 +2236,7 @@ pmap_remove_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_t sva, } else { ml3 = pmap_remove_pt_page(pmap, sva); if (ml3 != NULL) { - KASSERT(ml3->valid == VM_PAGE_BITS_ALL, + KASSERT(vm_page_any_valid(ml3), ("pmap_remove_l2: l3 page not promoted")); pmap_resident_count_dec(pmap, 1); KASSERT(ml3->ref_count == Ln_ENTRIES, @@ -2697,7 +2709,7 @@ pmap_demote_l2_locked(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, * If the page table page is not leftover from an earlier promotion, * initialize it. */ - if (mpte->valid == 0) { + if (!vm_page_all_valid(mpte)) { for (i = 0; i < Ln_ENTRIES; i++) pmap_store(firstl3 + i, newl3 + (i << PTE_PPN0_S)); } @@ -2706,8 +2718,7 @@ pmap_demote_l2_locked(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, "addresses")); /* - * If the mapping has changed attributes, update the page table - * entries. + * If the mapping has changed attributes, update the PTEs. */ if ((pmap_load(firstl3) & PTE_PROMOTE) != (newl3 & PTE_PROMOTE)) for (i = 0; i < Ln_ENTRIES; i++) @@ -2746,7 +2757,7 @@ static void pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t ml3, struct rwlock **lockp) { - pt_entry_t *firstl3, firstl3e, *l3, l3e; + pt_entry_t all_l3e_PTE_A, *firstl3, firstl3e, *l3, l3e; vm_paddr_t pa; PMAP_LOCK_ASSERT(pmap, MA_OWNED); @@ -2754,6 +2765,11 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t ml3, KASSERT((pmap_load(l2) & PTE_RWX) == 0, ("pmap_promote_l2: invalid l2 entry %p", l2)); + /* + * Examine the first L3E in the specified PTP. Abort if this L3E is + * ineligible for promotion or does not map the first 4KB physical page + * within a 2MB page. + */ firstl3 = (pt_entry_t *)PHYS_TO_DMAP(PTE_TO_PHYS(pmap_load(l2))); firstl3e = pmap_load(firstl3); pa = PTE_TO_PHYS(firstl3e); @@ -2781,8 +2797,14 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t ml3, } } - pa += PAGE_SIZE; - for (l3 = firstl3 + 1; l3 < firstl3 + Ln_ENTRIES; l3++) { + /* + * Examine each of the other PTEs in the specified PTP. Abort if this + * PTE maps an unexpected 4KB physical page or does not have identical + * characteristics to the first PTE. + */ + all_l3e_PTE_A = firstl3e & PTE_A; + pa += L2_SIZE - PAGE_SIZE; + for (l3 = firstl3 + Ln_ENTRIES - 1; l3 > firstl3; l3--) { l3e = pmap_load(l3); if (PTE_TO_PHYS(l3e) != pa) { CTR2(KTR_PMAP, @@ -2804,14 +2826,28 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t ml3, atomic_add_long(&pmap_l2_p_failures, 1); return; } - pa += PAGE_SIZE; + all_l3e_PTE_A &= l3e; + pa -= PAGE_SIZE; } + /* + * Unless all PTEs have PTE_A set, clear it from the superpage + * mapping, so that promotions triggered by speculative mappings, + * such as pmap_enter_quick(), don't automatically mark the + * underlying pages as referenced. + */ + firstl3e &= ~PTE_A | all_l3e_PTE_A; + + /* + * Save the page table page in its current state until the L2 + * mapping the superpage is demoted by pmap_demote_l2() or + * destroyed by pmap_remove_l3(). + */ if (ml3 == NULL) ml3 = PHYS_TO_VM_PAGE(PTE_TO_PHYS(pmap_load(l2))); KASSERT(ml3->pindex == pmap_l2_pindex(va), ("pmap_promote_l2: page table page's pindex is wrong")); - if (pmap_insert_pt_page(pmap, ml3, true)) { + if (pmap_insert_pt_page(pmap, ml3, true, all_l3e_PTE_A != 0)) { CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx pmap %p", va, pmap); atomic_add_long(&pmap_l2_p_failures, 1); @@ -3223,7 +3259,7 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t new_l2, u_int flags, * leave the kernel page table page zero filled. */ mt = PHYS_TO_VM_PAGE(PTE_TO_PHYS(pmap_load(l2))); - if (pmap_insert_pt_page(pmap, mt, false)) + if (pmap_insert_pt_page(pmap, mt, false, false)) panic("pmap_enter_l2: trie insert failed"); } else KASSERT(pmap_load(l2) == 0, @@ -3841,7 +3877,7 @@ pmap_remove_pages_pv(pmap_t pmap, vm_page_t m, pv_entry_t pv, } mpte = pmap_remove_pt_page(pmap, pv->pv_va); if (mpte != NULL) { - KASSERT(mpte->valid == VM_PAGE_BITS_ALL, + KASSERT(vm_page_any_valid(mpte), ("pmap_remove_pages: pte page not promoted")); pmap_resident_count_dec(pmap, 1); KASSERT(mpte->ref_count == Ln_ENTRIES,