From nobody Sat Jan 08 04:35:41 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D490519365B0; Sat, 8 Jan 2022 04:35:41 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JW6hs3TXTz3KHZ; Sat, 8 Jan 2022 04:35:41 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5734E1888D; Sat, 8 Jan 2022 04:35:41 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 2084ZfwX035995; Sat, 8 Jan 2022 04:35:41 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 2084ZfpG035994; Sat, 8 Jan 2022 04:35:41 GMT (envelope-from git) Date: Sat, 8 Jan 2022 04:35:41 GMT Message-Id: <202201080435.2084ZfpG035994@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Konstantin Belousov Subject: git: 720a892ac60f - main - amd64 pmap: simplify vtopte() and vtopde() List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 720a892ac60fc1fa7c784edc40f188c93ac7e0b7 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641616541; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=e5SJOzjFOIwREuT7mfhcWUaqqNV1m7fpUAlVB76QG+4=; b=UsE6U/5iYdo5SZrzkTgFyrYM7tNekvtTquJasIo50ezsTewTzSAD+yFKR9J2O/rWhyL/F4 6bOHcSvP5Jbep/cd3yEEd41/aYdqDqlHkSBRB2lMVoYnc99pBTC02VRn4Bs22XZ09TU5qo dlOUx5vg2JFu60ySQD4wEPBOC1evd8j+gPpP+ouFQNjDGr91BLFifPn6aS6FYvT4Zx37pZ S5jeAVZ5sdkQG6ZdXvcGkxVpk8y6ubn0P5ZGFGs6gjFh4Qhd1ijhSqCOa6uq9VOhN8E7FA MNxJ1Nnn8ZeTXqRO28CUsFzQyyXUfXm5map9vJFFSStx+8ZTgEK3nb9H7dyW9Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1641616541; a=rsa-sha256; cv=none; b=r1HjDQlw5aaOlbJNo6uMtboNhUUPxR8zrBs57OFWmCEFouxJ6a+mDjhNWvuCvDkaR7DwCW 2mBXTeg7SYJTFrVTmxy/vgqkzfyCqEa3e9EeSqmJui+bRkJ2WQUHw90IS7S7PAu38pjx3t vbrxHyMrN7f8bzfWuVCUyIyQuV7YZvzaJP32seRpFztPADhRF8XFDj96rTe3D5Z9dIY1ga Tze9dBND13bkP6kahNwenzzvm8gOD4gkf4SQ4UF8AYJhPo9PItzPhfTk39r3Y5HoGGLggG jDk0cMO51ADZz4rAxPRbI8tBWX2d0QXPwivpsldGZZVklnfOqQ2dDJNtcencMg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=720a892ac60fc1fa7c784edc40f188c93ac7e0b7 commit 720a892ac60fc1fa7c784edc40f188c93ac7e0b7 Author: Konstantin Belousov AuthorDate: 2022-01-05 00:26:12 +0000 Commit: Konstantin Belousov CommitDate: 2022-01-08 04:34:18 +0000 amd64 pmap: simplify vtopte() and vtopde() Pre-calculate masks and page table/page directory bases, for LA48 and LA57 cases, trading a conditional for the memory accesses. This shaves 672 bytes of .text, by the cost of 32 .data bytes. Note that eliminated code contained one conditional, and several costly movabs instructions, which are traded by two memory fetches per vtop{t,d}e() inlines. Reviewed by: markj Discussed with: alc Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D33776 --- sys/amd64/amd64/pmap.c | 39 +++++++++++++++++---------------------- 1 file changed, 17 insertions(+), 22 deletions(-) diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index a2d55b219505..8b5d58c2a540 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -1516,40 +1516,28 @@ pmap_pt_page_count_adj(pmap_t pmap, int count) } } +pt_entry_t vtoptem __read_mostly = (1ul << (NPTEPGSHIFT + NPDEPGSHIFT + + NPDPEPGSHIFT + NPML4EPGSHIFT)) - 1; +pt_entry_t *PTmap __read_mostly = P4Tmap; + PMAP_INLINE pt_entry_t * vtopte(vm_offset_t va) { - u_int64_t mask; - KASSERT(va >= VM_MAXUSER_ADDRESS, ("vtopte on a uva/gpa 0x%0lx", va)); - if (la57) { - mask = ((1ul << (NPTEPGSHIFT + NPDEPGSHIFT + NPDPEPGSHIFT + - NPML4EPGSHIFT + NPML5EPGSHIFT)) - 1); - return (P5Tmap + ((va >> PAGE_SHIFT) & mask)); - } else { - mask = ((1ul << (NPTEPGSHIFT + NPDEPGSHIFT + NPDPEPGSHIFT + - NPML4EPGSHIFT)) - 1); - return (P4Tmap + ((va >> PAGE_SHIFT) & mask)); - } + return (PTmap + ((va >> PAGE_SHIFT) & vtoptem)); } +pd_entry_t vtopdem __read_mostly = (1ul << (NPDEPGSHIFT + NPDPEPGSHIFT + + NPML4EPGSHIFT)) - 1; +pd_entry_t *PDmap __read_mostly = P4Dmap; + static __inline pd_entry_t * vtopde(vm_offset_t va) { - u_int64_t mask; - KASSERT(va >= VM_MAXUSER_ADDRESS, ("vtopde on a uva/gpa 0x%0lx", va)); - if (la57) { - mask = ((1ul << (NPDEPGSHIFT + NPDPEPGSHIFT + - NPML4EPGSHIFT + NPML5EPGSHIFT)) - 1); - return (P5Dmap + ((va >> PDRSHIFT) & mask)); - } else { - mask = ((1ul << (NPDEPGSHIFT + NPDPEPGSHIFT + - NPML4EPGSHIFT)) - 1); - return (P4Dmap + ((va >> PDRSHIFT) & mask)); - } + return (PDmap + ((va >> PDRSHIFT) & vtopdem)); } static u_int64_t @@ -2241,6 +2229,13 @@ pmap_bootstrap_la57(void *arg __unused) */ v_pml5[PML5PML5I] = KPML5phys | X86_PG_RW | X86_PG_V | pg_nx; + vtoptem = (1ul << (NPTEPGSHIFT + NPDEPGSHIFT + NPDPEPGSHIFT + + NPML4EPGSHIFT + NPML5EPGSHIFT)) - 1; + PTmap = P5Tmap; + vtopdem = (1ul << (NPDEPGSHIFT + NPDPEPGSHIFT + + NPML4EPGSHIFT + NPML5EPGSHIFT)) - 1; + PDmap = P5Dmap; + kernel_pmap->pm_cr3 = KPML5phys; kernel_pmap->pm_pmltop = v_pml5; pmap_pt_page_count_adj(kernel_pmap, 1);