From nobody Sat Aug 02 17:15:01 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bvTw52GSmz63Lby; Sat, 02 Aug 2025 17:15:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bvTw51GSQz3GWs; Sat, 02 Aug 2025 17:15:01 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1754154901; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=wmPSxCTiy8tiJXFn5jk0rhTLYIiYdl+/MWY8wECmq90=; b=yB3TfazDxc4l5LBgmpc7B32txEZfMxe90i1XxfC3teir+O8vMejYFC8cUBiaxhB2vo3ANr 45qDtJ+ieXWy/XOfKCfi70MSPZtyEdPPvU63DZKU3u0ID+EpH5LDeif72agAoyc2vWa1Eb Dt3ESEVOURkR16n+v3DE1OtREeKPQZBx4RoEt4wm7szWYKAxpBm0Ty4VEbF4YeDkGOdSdB i1TRC/S6UcrriuMjD5jcNVR7HBfOenfyeHx08EBt+Ff+T9+mrljTY0IueS7bIGQVJELJ8n 86nGJcje27eVvOPqqjmTl/C7L33tj5v3a0MtZcwgRNTo7GjVo4V0lyzpQ1lx0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1754154901; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=wmPSxCTiy8tiJXFn5jk0rhTLYIiYdl+/MWY8wECmq90=; b=frDmIw1dxVSndnjDQ32nR8AG1N/nt2AP3Vu4JzGip4MdWMt8a85QXe2KidhOzDFrUPUd24 QoBshG8FBolEg+xErtNqnVzXQ/5EXtqPhQvk/02gvMCTKEeUnIdS7tI4XsgF8HYfZ6sY36 lemUurbyebJUB4sCy6jflTzeptoz5DobKjrwZYlOQSVW0sn4jZLexRg3B5ZuLisMJIghVH klwdIhnsQT9RlX6ZvVRn3nDmzvWtL7zkIgaA8WnYVXUg55FIrYezdPWrud+OpHBhqM26th Pyhint9fysYmeDdeJQcePKPuK5g4klr0meNrKRuYt0gYoqD6bRnK4OMYF9LEJA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1754154901; a=rsa-sha256; cv=none; b=PcD/O0Iw+GAirTiIOSa8qLRkRtDC+aZ1kQ2g1qz4+jXyE1i2pToC4agKX4W/IWy+q20zuC y50UZlxKW2oeEMw0MMybI6QCFM++hbl+WjIzeEt833QO1ksAoRTlFW9XMpRYL5ZOIuGiqO 46qwi4W2neLtdJciXK+B+6OafAvIDIIQBfxCGNcoDyob+T7SNSQ/sAlraDSbNZiMiU8ev3 IsOrz78YxROejOCkZqfAgDHLcobDTjP1RrZyxb+Y9VdTQwX6Tv0m9ZBS58W+WgaS/RUJDf GUoztNXned8c6GLswUZBhz0Cy6kVrU6A29OwgupqJgXrG3ZEDmANBi3xto/9nA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4bvTw50sKJzBdc; Sat, 02 Aug 2025 17:15:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 572HF1Rp091037; Sat, 2 Aug 2025 17:15:01 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 572HF1Q7091034; Sat, 2 Aug 2025 17:15:01 GMT (envelope-from git) Date: Sat, 2 Aug 2025 17:15:01 GMT Message-Id: <202508021715.572HF1Q7091034@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: 6fb848f2ff91 - main - amd64 pmap: Use INVPCID_CTXGLOB on Ryzen processors List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6fb848f2ff91337dbb26024ab41103ee2b036021 Auto-Submitted: auto-generated The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=6fb848f2ff91337dbb26024ab41103ee2b036021 commit 6fb848f2ff91337dbb26024ab41103ee2b036021 Author: Alan Cox AuthorDate: 2025-07-07 02:45:11 +0000 Commit: Alan Cox CommitDate: 2025-08-02 17:14:39 +0000 amd64 pmap: Use INVPCID_CTXGLOB on Ryzen processors Recent AMD Ryzen processors support a limited form of the invpcid instruction, even when they do not support PCID functionality. In particular, they support the type 2 form of the instruction, what we call INVPCID_CTXGLOB. This is faster than toggling PGE in cr4. Reviewed by: kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D51565 --- sys/amd64/amd64/machdep.c | 7 ++--- sys/amd64/amd64/pmap.c | 76 +++++++++++++++++++++++++++-------------------- 2 files changed, 45 insertions(+), 38 deletions(-) diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index 37c7056f649c..9ff60439d1ec 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -1336,12 +1336,9 @@ hammer_time(u_int64_t modulep, u_int64_t physfree) pti = pti_get_default(); TUNABLE_INT_FETCH("vm.pmap.pti", &pti); TUNABLE_INT_FETCH("vm.pmap.pcid_enabled", &pmap_pcid_enabled); - if ((cpu_feature2 & CPUID2_PCID) != 0 && pmap_pcid_enabled) { - invpcid_works = (cpu_stdext_feature & - CPUID_STDEXT_INVPCID) != 0; - } else { + if ((cpu_feature2 & CPUID2_PCID) == 0) pmap_pcid_enabled = 0; - } + invpcid_works = (cpu_stdext_feature & CPUID_STDEXT_INVPCID) != 0; /* * Now we can do small core initialization, after the PCID diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index 6369498210d9..243a6625bece 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -2904,6 +2904,9 @@ pmap_update_pde_invalidate(pmap_t pmap, vm_offset_t va, pd_entry_t newpde) /* * Promotion: flush every 4KB page mapping from the TLB, * including any global (PG_G) mappings. + * + * This function is only used on older processors that + * do not support the invpcid instruction. */ invltlb_glob(); } @@ -3050,13 +3053,13 @@ pmap_update_pde_invalidate(pmap_t pmap, vm_offset_t va, pd_entry_t newpde) * local user page: INVLPG * local kernel page: INVLPG * local user total: reload %cr3 - * local kernel total: invltlb_glob() + * local kernel total: INVPCID(CTXGLOB) or invltlb_glob() * remote user page, inactive pmap: - * remote user page, active pmap: IPI:INVLPG * remote kernel page: IPI:INVLPG * remote user total, inactive pmap: - * remote user total, active pmap: IPI:(reload %cr3) - * remote kernel total: IPI:invltlb_glob() + * remote kernel total: IPI:INVPCID(CTXGLOB) or invltlb_glob() * Since on return to user mode, the reload of %cr3 with ucr3 causes * TLB invalidation, no specific action is required for user page table. * @@ -3356,7 +3359,8 @@ pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) } static inline void -pmap_invalidate_all_pcid_cb(pmap_t pmap, bool invpcid_works1) +pmap_invalidate_all_cb_template(pmap_t pmap, bool pmap_pcid_enabled1, + bool invpcid_works1) { struct invpcid_descr d; uint64_t kcr3; @@ -3370,57 +3374,63 @@ pmap_invalidate_all_pcid_cb(pmap_t pmap, bool invpcid_works1) invltlb_glob(); } } else if (pmap == PCPU_GET(curpmap)) { - CRITICAL_ASSERT(curthread); - - pcid = pmap_get_pcid(pmap); - if (invpcid_works1) { - d.pcid = pcid; - d.pad = 0; - d.addr = 0; - invpcid(&d, INVPCID_CTX); + if (pmap_pcid_enabled1) { + CRITICAL_ASSERT(curthread); + + pcid = pmap_get_pcid(pmap); + if (invpcid_works1) { + d.pcid = pcid; + d.pad = 0; + d.addr = 0; + invpcid(&d, INVPCID_CTX); + } else { + kcr3 = pmap->pm_cr3 | pcid; + load_cr3(kcr3); + } + if (pmap->pm_ucr3 != PMAP_NO_CR3) + PCPU_SET(ucr3_load_mask, ~CR3_PCID_SAVE); } else { - kcr3 = pmap->pm_cr3 | pcid; - load_cr3(kcr3); + invltlb(); } - if (pmap->pm_ucr3 != PMAP_NO_CR3) - PCPU_SET(ucr3_load_mask, ~CR3_PCID_SAVE); } } static void -pmap_invalidate_all_pcid_invpcid_cb(pmap_t pmap) +pmap_invalidate_all_pcid_invpcid_cb(pmap_t pmap, vm_offset_t addr1 __unused, + vm_offset_t addr2 __unused) { - pmap_invalidate_all_pcid_cb(pmap, true); + pmap_invalidate_all_cb_template(pmap, true, true); } static void -pmap_invalidate_all_pcid_noinvpcid_cb(pmap_t pmap) +pmap_invalidate_all_pcid_noinvpcid_cb(pmap_t pmap, vm_offset_t addr1 __unused, + vm_offset_t addr2 __unused) { - pmap_invalidate_all_pcid_cb(pmap, false); + pmap_invalidate_all_cb_template(pmap, true, false); } static void -pmap_invalidate_all_nopcid_cb(pmap_t pmap) +pmap_invalidate_all_nopcid_invpcid_cb(pmap_t pmap, vm_offset_t addr1 __unused, + vm_offset_t addr2 __unused) { - if (pmap == kernel_pmap) - invltlb_glob(); - else if (pmap == PCPU_GET(curpmap)) - invltlb(); + pmap_invalidate_all_cb_template(pmap, false, true); } -DEFINE_IFUNC(static, void, pmap_invalidate_all_cb, (pmap_t)) +static void +pmap_invalidate_all_nopcid_noinvpcid_cb(pmap_t pmap, vm_offset_t addr1 __unused, + vm_offset_t addr2 __unused) { - if (pmap_pcid_enabled) - return (invpcid_works ? pmap_invalidate_all_pcid_invpcid_cb : - pmap_invalidate_all_pcid_noinvpcid_cb); - return (pmap_invalidate_all_nopcid_cb); + pmap_invalidate_all_cb_template(pmap, false, false); } -static void -pmap_invalidate_all_curcpu_cb(pmap_t pmap, vm_offset_t addr1 __unused, - vm_offset_t addr2 __unused) +DEFINE_IFUNC(static, void, pmap_invalidate_all_curcpu_cb, (pmap_t, vm_offset_t, + vm_offset_t)) { - pmap_invalidate_all_cb(pmap); + if (pmap_pcid_enabled) + return (invpcid_works ? pmap_invalidate_all_pcid_invpcid_cb : + pmap_invalidate_all_pcid_noinvpcid_cb); + return (invpcid_works ? pmap_invalidate_all_nopcid_invpcid_cb : + pmap_invalidate_all_nopcid_noinvpcid_cb); } void