From nobody Tue May 02 11:39:53 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9dRF3sZNz48tLy; Tue, 2 May 2023 11:39:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9dRF2yHyz3qHh; Tue, 2 May 2023 11:39:53 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683027593; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tsjRBY6jJrr88Ej/fXla6xr34i14eivdEwfnoCxXACo=; b=Dnk9wmaJYnVAEhRLn6QMR4eJVDevpPg0QCdb+i3koHWFBIlCu9tuQ58iUigziLVWZVeRoH fuFP8YCJZgUykDPjQwAFkye+j7DKUwTrmY4RxSGclmiknbtbPfZuK4YH1efFNWCSMRR44i Yw8DdlQ6s9rWPegLIEd7lG5ftTVOUrsZyULejCzmIOPtpbl7WwsvxS9BqTC3UrX71mGf2k 8XWuoDitEPbYr2TzbWNzS/vdMnzrJn7hPgMIlaYX6d96J6Ow1yzOXA4A9GOYRRonm4q8j/ 5mm3bSBmniy/CF9X7/Rh8JsMYKMxQKWyin3EHOze1Dv08hE4Ck96hPBH4i90BA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683027593; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tsjRBY6jJrr88Ej/fXla6xr34i14eivdEwfnoCxXACo=; b=vWe2FrxOHBOG7fHiLU6Gc2tM8L0JpnTBFBHi9Pbo8sor+OdEWtWwZP5f5lSby7Evo2nvz8 aETjJ7NihgJda+ajYmWGqI+vZDIZCDSguwY/skd399CJ3Mh5qQWB94ffq1Kiik5z8wRdIT ewF4nZScjjcCWRbp/mTbfMvV28j1lE33RWU2D8bbYi7rGyHGcSiRMKhc8suTo+DIinbAcn chse+FduQGERH5glWRZtzk5T/TeNPp/t7YeT4lm9TUAl4tnw9aSoqliUssoyqkhZpf8tx5 FbnBTW0tEhiyGrX/yDiJulPlgASW/UK7n28o2TW9DfARnwOVDSW5CH7GLfaSRg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1683027593; a=rsa-sha256; cv=none; b=AcRxhIsJNQfnDpuneMwv+9p/JBqEJaMXC3Pv0jL1A77qbcLotgm0hk8l9k9GnC8r8Xd9aR TamoUfWtxorQukUb+aLHbyIQwCTJTIZTl66BwejHRFTvkBXP2WO/uoyPtX4HOXBUbtaB/h bssSPzaoazyXN2ML/qUzPNHR3lPdVFSDbw3ZNKMZM3kxZwbt9nb6c1x8Rrk2lXm4tzRwQ/ JP+ZI2el0oB+wmV+DU7yrgkzMJuStBaj50ThsRAIjqJ2RJPwIpoZZisPuimbwZjvLfs/j3 q5Nf5Y6wOY79FeQL9foOG8lzKa5X+93KXbOkW1UvLqT9w7QzsWi7prqyB9KNrw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Q9dRF1wcTzmB1; Tue, 2 May 2023 11:39:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 342Bdrea092324; Tue, 2 May 2023 11:39:53 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 342BdrSS092323; Tue, 2 May 2023 11:39:53 GMT (envelope-from git) Date: Tue, 2 May 2023 11:39:53 GMT Message-Id: <202305021139.342BdrSS092323@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Konstantin Belousov Subject: git: 9e0143694ad3 - main - amd64: add pmap_get_pcid() helper List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 9e0143694ad31b534310a473b993e520264f11e2 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=9e0143694ad31b534310a473b993e520264f11e2 commit 9e0143694ad31b534310a473b993e520264f11e2 Author: Konstantin Belousov AuthorDate: 2023-05-01 11:02:59 +0000 Commit: Konstantin Belousov CommitDate: 2023-05-02 11:32:35 +0000 amd64: add pmap_get_pcid() helper Reviewed by: markj Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D39890 --- sys/amd64/amd64/efirt_machdep.c | 19 +++++++++++++------ sys/amd64/amd64/mp_machdep.c | 20 ++++++++------------ sys/amd64/amd64/pmap.c | 16 ++++------------ sys/amd64/include/pmap.h | 8 ++++++++ 4 files changed, 33 insertions(+), 30 deletions(-) diff --git a/sys/amd64/amd64/efirt_machdep.c b/sys/amd64/amd64/efirt_machdep.c index 6fffd113f868..b7f7a91b8671 100644 --- a/sys/amd64/amd64/efirt_machdep.c +++ b/sys/amd64/amd64/efirt_machdep.c @@ -298,6 +298,7 @@ int efi_arch_enter(void) { pmap_t curpmap; + uint64_t cr3; curpmap = PCPU_GET(curpmap); PMAP_LOCK_ASSERT(curpmap, MA_OWNED); @@ -313,8 +314,10 @@ efi_arch_enter(void) if (pmap_pcid_enabled && !invpcid_works) PCPU_SET(curpmap, NULL); - load_cr3(VM_PAGE_TO_PHYS(efi_pmltop_page) | (pmap_pcid_enabled ? - curpmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid : 0)); + cr3 = VM_PAGE_TO_PHYS(efi_pmltop_page); + if (pmap_pcid_enabled) + cr3 |= pmap_get_pcid(curpmap); + load_cr3(cr3); /* * If PCID is enabled, the clear CR3_PCID_SAVE bit in the loaded %cr3 * causes TLB invalidation. @@ -328,12 +331,16 @@ void efi_arch_leave(void) { pmap_t curpmap; + uint64_t cr3; curpmap = &curproc->p_vmspace->vm_pmap; - if (pmap_pcid_enabled && !invpcid_works) - PCPU_SET(curpmap, curpmap); - load_cr3(curpmap->pm_cr3 | (pmap_pcid_enabled ? - curpmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid : 0)); + cr3 = curpmap->pm_cr3; + if (pmap_pcid_enabled) { + cr3 |= pmap_get_pcid(curpmap); + if (!invpcid_works) + PCPU_SET(curpmap, curpmap); + } + load_cr3(cr3); if (!pmap_pcid_enabled) invltlb(); vm_fault_enable_pagefaults(curthread->td_md.md_efirt_dis_pf); diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c index 5c60d301c1e7..94589f736637 100644 --- a/sys/amd64/amd64/mp_machdep.c +++ b/sys/amd64/amd64/mp_machdep.c @@ -767,7 +767,7 @@ invltlb_invpcid_handler(pmap_t smp_tlb_pmap) (*ipi_invltlb_counts[PCPU_GET(cpuid)])++; #endif /* COUNT_IPIS */ - d.pcid = smp_tlb_pmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid; + d.pcid = pmap_get_pcid(smp_tlb_pmap); d.pad = 0; d.addr = 0; invpcid(&d, smp_tlb_pmap == kernel_pmap ? INVPCID_CTXGLOB : @@ -786,7 +786,7 @@ invltlb_invpcid_pti_handler(pmap_t smp_tlb_pmap) (*ipi_invltlb_counts[PCPU_GET(cpuid)])++; #endif /* COUNT_IPIS */ - d.pcid = smp_tlb_pmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid; + d.pcid = pmap_get_pcid(smp_tlb_pmap); d.pad = 0; d.addr = 0; if (smp_tlb_pmap == kernel_pmap) { @@ -808,8 +808,6 @@ invltlb_invpcid_pti_handler(pmap_t smp_tlb_pmap) static void invltlb_pcid_handler(pmap_t smp_tlb_pmap) { - uint32_t pcid; - #ifdef COUNT_XINVLTLB_HITS xhits_gbl[PCPU_GET(cpuid)]++; #endif /* COUNT_XINVLTLB_HITS */ @@ -828,8 +826,8 @@ invltlb_pcid_handler(pmap_t smp_tlb_pmap) * CPU. */ if (smp_tlb_pmap == PCPU_GET(curpmap)) { - pcid = smp_tlb_pmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid; - load_cr3(smp_tlb_pmap->pm_cr3 | pcid); + load_cr3(smp_tlb_pmap->pm_cr3 | + pmap_get_pcid(smp_tlb_pmap)); if (smp_tlb_pmap->pm_ucr3 != PMAP_NO_CR3) PCPU_SET(ucr3_load_mask, ~CR3_PCID_SAVE); } @@ -865,8 +863,7 @@ invlpg_invpcid_handler(pmap_t smp_tlb_pmap, vm_offset_t smp_tlb_addr1) if (smp_tlb_pmap == PCPU_GET(curpmap) && smp_tlb_pmap->pm_ucr3 != PMAP_NO_CR3 && PCPU_GET(ucr3_load_mask) == PMAP_UCR3_NOMASK) { - d.pcid = smp_tlb_pmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid | - PMAP_PCID_USER_PT; + d.pcid = pmap_get_pcid(smp_tlb_pmap) | PMAP_PCID_USER_PT; d.pad = 0; d.addr = smp_tlb_addr1; invpcid(&d, INVPCID_ADDR); @@ -890,7 +887,7 @@ invlpg_pcid_handler(pmap_t smp_tlb_pmap, vm_offset_t smp_tlb_addr1) if (smp_tlb_pmap == PCPU_GET(curpmap) && (ucr3 = smp_tlb_pmap->pm_ucr3) != PMAP_NO_CR3 && PCPU_GET(ucr3_load_mask) == PMAP_UCR3_NOMASK) { - pcid = smp_tlb_pmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid; + pcid = pmap_get_pcid(smp_tlb_pmap); kcr3 = smp_tlb_pmap->pm_cr3 | pcid | CR3_PCID_SAVE; ucr3 |= pcid | PMAP_PCID_USER_PT | CR3_PCID_SAVE; pmap_pti_pcid_invlpg(ucr3, kcr3, smp_tlb_addr1); @@ -944,8 +941,7 @@ invlrng_invpcid_handler(pmap_t smp_tlb_pmap, vm_offset_t smp_tlb_addr1, if (smp_tlb_pmap == PCPU_GET(curpmap) && smp_tlb_pmap->pm_ucr3 != PMAP_NO_CR3 && PCPU_GET(ucr3_load_mask) == PMAP_UCR3_NOMASK) { - d.pcid = smp_tlb_pmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid | - PMAP_PCID_USER_PT; + d.pcid = pmap_get_pcid(smp_tlb_pmap) | PMAP_PCID_USER_PT; d.pad = 0; d.addr = smp_tlb_addr1; do { @@ -978,7 +974,7 @@ invlrng_pcid_handler(pmap_t smp_tlb_pmap, vm_offset_t smp_tlb_addr1, if (smp_tlb_pmap == PCPU_GET(curpmap) && (ucr3 = smp_tlb_pmap->pm_ucr3) != PMAP_NO_CR3 && PCPU_GET(ucr3_load_mask) == PMAP_UCR3_NOMASK) { - pcid = smp_tlb_pmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid; + pcid = pmap_get_pcid(smp_tlb_pmap); kcr3 = smp_tlb_pmap->pm_cr3 | pcid | CR3_PCID_SAVE; ucr3 |= pcid | PMAP_PCID_USER_PT | CR3_PCID_SAVE; pmap_pti_pcid_invlrng(ucr3, kcr3, smp_tlb_addr1, smp_tlb_addr2); diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index 3b125041bb29..16bfe9986c6b 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -3076,7 +3076,6 @@ pmap_invalidate_page_pcid_cb(pmap_t pmap, vm_offset_t va, struct invpcid_descr d; uint64_t kcr3, ucr3; uint32_t pcid; - u_int cpuid; /* * Because pm_pcid is recalculated on a context switch, we @@ -3095,9 +3094,7 @@ pmap_invalidate_page_pcid_cb(pmap_t pmap, vm_offset_t va, PCPU_GET(ucr3_load_mask) != PMAP_UCR3_NOMASK) return; - cpuid = PCPU_GET(cpuid); - - pcid = pmap->pm_pcids[cpuid].pm_pcid; + pcid = pmap_get_pcid(pmap); if (invpcid_works1) { d.pcid = pcid | PMAP_PCID_USER_PT; d.pad = 0; @@ -3172,7 +3169,6 @@ pmap_invalidate_range_pcid_cb(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, struct invpcid_descr d; uint64_t kcr3, ucr3; uint32_t pcid; - u_int cpuid; CRITICAL_ASSERT(curthread); @@ -3181,9 +3177,7 @@ pmap_invalidate_range_pcid_cb(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, PCPU_GET(ucr3_load_mask) != PMAP_UCR3_NOMASK) return; - cpuid = PCPU_GET(cpuid); - - pcid = pmap->pm_pcids[cpuid].pm_pcid; + pcid = pmap_get_pcid(pmap); if (invpcid_works1) { d.pcid = pcid | PMAP_PCID_USER_PT; d.pad = 0; @@ -3273,7 +3267,6 @@ pmap_invalidate_all_pcid_cb(pmap_t pmap, bool invpcid_works1) struct invpcid_descr d; uint64_t kcr3; uint32_t pcid; - u_int cpuid; if (pmap == kernel_pmap) { if (invpcid_works1) { @@ -3284,9 +3277,8 @@ pmap_invalidate_all_pcid_cb(pmap_t pmap, bool invpcid_works1) } } else if (pmap == PCPU_GET(curpmap)) { CRITICAL_ASSERT(curthread); - cpuid = PCPU_GET(cpuid); - pcid = pmap->pm_pcids[cpuid].pm_pcid; + pcid = pmap_get_pcid(pmap); if (invpcid_works1) { d.pcid = pcid; d.pad = 0; @@ -10152,7 +10144,7 @@ pmap_activate_boot(pmap_t pmap) if (pti) { kcr3 = pmap->pm_cr3; if (pmap_pcid_enabled) - kcr3 |= pmap->pm_pcids[cpuid].pm_pcid | CR3_PCID_SAVE; + kcr3 |= pmap_get_pcid(pmap) | CR3_PCID_SAVE; } else { kcr3 = PMAP_NO_CR3; } diff --git a/sys/amd64/include/pmap.h b/sys/amd64/include/pmap.h index 3b99b88742e5..cef718fb50d5 100644 --- a/sys/amd64/include/pmap.h +++ b/sys/amd64/include/pmap.h @@ -533,6 +533,14 @@ pmap_invlpg(pmap_t pmap, vm_offset_t va) } #endif /* sys/pcpu.h && machine/cpufunc.h */ +/* Return pcid for the pmap pmap on current cpu */ +static __inline uint32_t +pmap_get_pcid(pmap_t pmap) +{ + MPASS(pmap_pcid_enabled); + return (pmap->pm_pcids[PCPU_GET(cpuid)].pm_pcid); +} + #endif /* _KERNEL */ /* Return various clipped indexes for a given VA */