From nobody Wed Mar 30 21:04:38 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0A1841A3F7B1; Wed, 30 Mar 2022 21:04:39 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KTJpZ6s0Gz4Vmd; Wed, 30 Mar 2022 21:04:38 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648674279; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=VlcK2hViWm/b+juObNWKqfgzQKoaKrI15uRkpEvJgAc=; b=B4WGEh1Um7AW4DYy6T6IKWH/i2ulm3IvFXBLEqqnyLSgsxAFGc5bfwUmqlyJlSDIrJRRSQ xOxwFyDoG5eCt2/cNl2IGgPzdYA/ouGzUZq8Gk6ds/0TMNUnxiY4bn7NlKBs18RdpItIfg Oz+TAZ6TWFNtOFjG/n+CxEKA2gVuhxFBueiKq2QHg70RlqZwLdoIsVoBKbGOpNzXf/jxjZ RYNS3lZ+jJvHv+DAwLD2Sh4oaFoSGNReWGBSEKOVtNCB1xKlnAPARzeeifmwiA0EG5VXG4 Zyr44gZ5VbWtTWePhaVhhF7KAaaGo9nN51XyAKbXJCIQl7OpzN3fKr6inU4vKA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C0E151A9F7; Wed, 30 Mar 2022 21:04:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 22UL4c4I052958; Wed, 30 Mar 2022 21:04:38 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 22UL4chb052957; Wed, 30 Mar 2022 21:04:38 GMT (envelope-from git) Date: Wed, 30 Mar 2022 21:04:38 GMT Message-Id: <202203302104.22UL4chb052957@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: D Scott Phillips Subject: git: cf9f384de949 - releng/13.1 - Fix arm64 TLB invalidation with non-4k pages List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: scottph X-Git-Repository: src X-Git-Refname: refs/heads/releng/13.1 X-Git-Reftype: branch X-Git-Commit: cf9f384de949b34f2ce73ca2f2f5e319431d46ff Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648674279; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=VlcK2hViWm/b+juObNWKqfgzQKoaKrI15uRkpEvJgAc=; b=cYopUBdPOqCVcHeJ2xyLamI1vczyGTSFwkuCIgmcf87g7POlRvWvJ24p6SRiTPqSH7QFH0 tmWOdgPnrHu0qGWwTIgMOQcVDIRvuKncDOkt66ziWeS1RAjgcArcUoEsC9mBL9jnuLv4Nv ZoSc5BtaV7jQWtz5dD822bx2J/awUbfFKq1SXIZsz6lsrZfveKvpDSRYCFlnpn+RxbauA+ uQUGu3kmoX20DEdB6f4yuMSxBOHQIr9V22sogzKxviHK68OzbNWDBQHz8B0jfgzMdtYpZs CcO8sR8yoALfxBAVH9QLKZhLa5eKYDyfDgItz3fvkXUb990Rc0mbqRQaqkjLRw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648674279; a=rsa-sha256; cv=none; b=TpTweIoe72cUEgSUfIC2EHQAkpc7wcFLtdeXlSJsXQ7yBhyhiubLf5/Y5LcWTmgivmdIGu siT0PbKboQZ2BmtSYDm7zK8sfHSQrM6dYKYO97/OPC2TNnumQ2caE/SUf/QapKd66RWj1n Xo8ZXJpRxo4wEDijZ8fqKtrk46S+UmEqEylW2IxahPe2kz6vytWkKescvE4GVqmQehsjIx L57FM5e9JJQs3A8aIpjsqQb1iahKyF2iW4tXpMu828lmtLca+AymXgB8NuALTAqusPfmXE 3Pq8mdklu9Wn0acNOWzyUaZsIPFiBAumO7UM5AbZn1sJEeMwlLXYr6qnqeeekw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch releng/13.1 has been updated by scottph: URL: https://cgit.FreeBSD.org/src/commit/?id=cf9f384de949b34f2ce73ca2f2f5e319431d46ff commit cf9f384de949b34f2ce73ca2f2f5e319431d46ff Author: Andrew Turner AuthorDate: 2022-03-10 14:39:03 +0000 Commit: D Scott Phillips CommitDate: 2022-03-30 20:50:58 +0000 Fix arm64 TLB invalidation with non-4k pages When using 16k or 64k pages atop will shift the address by more than the needed amount for a tlbi instruction. Replace this with a new macro to shift the address by 12 and use PAGE_SIZE in the for loop to let the code work with any page size. Approved by: re (gjb) Reviewed by: alc, markj Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D34516 (cherry picked from commit 813738fabaaea43503724b8371faf5bab73a3047) (cherry picked from commit 89166c063a9a136ba3780bb069e03e6049c53fd5) --- sys/arm64/arm64/pmap.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index c105e7cae16b..26e62c9e3cbb 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -360,6 +360,10 @@ void (*pmap_invalidate_vpipt_icache)(void); #define COOKIE_TO_ASID(cookie) ((int)(cookie)) #define COOKIE_TO_EPOCH(cookie) ((int)((u_long)(cookie) >> 32)) +#define TLBI_VA_SHIFT 12 +#define TLBI_VA(addr) ((addr) >> TLBI_VA_SHIFT) +#define TLBI_VA_L3_INCR (L3_SIZE >> TLBI_VA_SHIFT) + static int superpages_enabled = 1; SYSCTL_INT(_vm_pmap, OID_AUTO, superpages_enabled, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &superpages_enabled, 0, @@ -1186,11 +1190,11 @@ pmap_invalidate_page(pmap_t pmap, vm_offset_t va) PMAP_ASSERT_STAGE1(pmap); dsb(ishst); + r = TLBI_VA(va); if (pmap == kernel_pmap) { - r = atop(va); __asm __volatile("tlbi vaae1is, %0" : : "r" (r)); } else { - r = ASID_TO_OPERAND(COOKIE_TO_ASID(pmap->pm_cookie)) | atop(va); + r |= ASID_TO_OPERAND(COOKIE_TO_ASID(pmap->pm_cookie)); __asm __volatile("tlbi vae1is, %0" : : "r" (r)); } dsb(ish); @@ -1206,15 +1210,15 @@ pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) dsb(ishst); if (pmap == kernel_pmap) { - start = atop(sva); - end = atop(eva); - for (r = start; r < end; r++) + start = TLBI_VA(sva); + end = TLBI_VA(eva); + for (r = start; r < end; r += TLBI_VA_L3_INCR) __asm __volatile("tlbi vaae1is, %0" : : "r" (r)); } else { start = end = ASID_TO_OPERAND(COOKIE_TO_ASID(pmap->pm_cookie)); - start |= atop(sva); - end |= atop(eva); - for (r = start; r < end; r++) + start |= TLBI_VA(sva); + end |= TLBI_VA(eva); + for (r = start; r < end; r += TLBI_VA_L3_INCR) __asm __volatile("tlbi vae1is, %0" : : "r" (r)); } dsb(ish);