From nobody Tue Jun 07 14:23:31 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A67347CAB7E; Tue, 7 Jun 2022 14:23:31 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LHXdv3MHXz3rHW; Tue, 7 Jun 2022 14:23:31 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1654611811; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FyCZAixoPvHb4DLWTjwGQBMaxPIqII4cuFgZTs2lcUU=; b=oh5eTgs3yv9Xls/1OUxtBN457LDkNkhjXbKznA96BTKttperf/VXz94S8sqmzAACMVfUJ5 9nBPhGTy7vxeeDXqJdRX9RNdaxImhwaibZiR0yOES4Sc02man33IOtIlLABfsw082sJrwU H2yeraqhSaAAYwqnhz7/TWv3LW7ziahbIedM5xv9hkqpfZ3t+mLo/AO0KU4FE9Sg6trtgU C/rNw4ryijhEXLxGZTVbJYj4KGl1saqz+vxghjebkFVrMVRnon9iZuXR55bqPGXpbWPW0J 9EnEDz/CC5sbXNWV+aTpEhaJnKiXf/HZdWI4mEQeh1hc6AFuMtwdg+iRibplMg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 305B71C9AB; Tue, 7 Jun 2022 14:23:31 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 257ENV3O020332; Tue, 7 Jun 2022 14:23:31 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 257ENVuP020331; Tue, 7 Jun 2022 14:23:31 GMT (envelope-from git) Date: Tue, 7 Jun 2022 14:23:31 GMT Message-Id: <202206071423.257ENVuP020331@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Andrew Turner Subject: git: 20d21e435145 - stable/13 - Support LSE atomics in the arm64 casue* functions List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: andrew X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 20d21e43514502c0a346f3c2cdb1d8cd8c7cc6b0 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1654611811; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FyCZAixoPvHb4DLWTjwGQBMaxPIqII4cuFgZTs2lcUU=; b=tA4vEoehSpAYUuabxxMWciQadomUduGJX9HfQvl3T4+0HUd7DqOvEIh2V83m39ghC3IlEp yt7tY6MgkpbACjNXJ9uRmalqe2K1WpFeiEz8GIWv6LYIlNr5GCrqNNOxE6vDT5FnlM/kuW 9+DiSnX1UBJ8pnYIHbod7lyjpfcJTCahrFVh/WnEPbpk/+7SuC21ZLcpsHQv0WDu3AtS7y uff4LFk+HFsmr0Cl9rmij58dR3e2oNoIvBaXr8dcIDJBXg9ZL7BgI7fgLOTaksVTYMy/C8 3idl4hDym3zlhh7RD7BzVbZHVx/l6zPBjEuDwkzPq3c58wd3t78gvo1DHcwDCw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1654611811; a=rsa-sha256; cv=none; b=WkTCkfDu2ZiS84NDpQUMf1sRpBguPLl2oY5MBMWiJ1yq/sYM5SMjKzgyGdCA4CdoexVflB 888ySIboqZHV0OeIa8hir2EZFdoN7f7kB/Y9cck5pSJVRIYRy66z07V0Pa/qxoAjkj66h3 AlXKEMMBAJ3jMKquhn+zAZFi20WnP/DxyQ1wlLoPwqD9X72Bb+w7LmtIq+SXKej0RVP2Ue UEPWt3K1b4cosMZn9aRHojvWoh+ZH7tW+wV3H8KnMAtiBgUz+5RaCdGifBVoBzJyGxhiTw Dkj28pP+vSK3NltF8welzlR0XI6MnUf2uutefDD5l1kWtEd+HPuFf6Fx1jnXcg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by andrew: URL: https://cgit.FreeBSD.org/src/commit/?id=20d21e43514502c0a346f3c2cdb1d8cd8c7cc6b0 commit 20d21e43514502c0a346f3c2cdb1d8cd8c7cc6b0 Author: Andrew Turner AuthorDate: 2022-05-17 13:52:30 +0000 Commit: Andrew Turner CommitDate: 2022-06-07 14:20:18 +0000 Support LSE atomics in the arm64 casue* functions As with atomic(9) use the ARMv8.1 Large System Extension atomic instructions to implement the userspace compare and swap functions. Reviewed by: kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D35234 (cherry picked from commit 88ac318759f28f34aa991ecc6f336326ec00c501) --- sys/arm64/arm64/support.S | 52 +++++++++++++++++++++++++++++++----- sys/arm64/arm64/support_ifunc.c | 58 +++++++++++++++++++++++++++++++++++++++++ sys/conf/files.arm64 | 1 + 3 files changed, 105 insertions(+), 6 deletions(-) diff --git a/sys/arm64/arm64/support.S b/sys/arm64/arm64/support.S index 4771814a9b45..885af5676159 100644 --- a/sys/arm64/arm64/support.S +++ b/sys/arm64/arm64/support.S @@ -56,9 +56,9 @@ fsu_fault_nopcb: END(fsu_fault) /* - * int casueword32(volatile uint32_t *, uint32_t, uint32_t *, uint32_t) + * int casueword32_llsc(volatile uint32_t *, uint32_t, uint32_t *, uint32_t) */ -ENTRY(casueword32) +ENTRY(casueword32_llsc) check_user_access 0, (VM_MAXUSER_ADDRESS-3), fsu_fault_nopcb adr x6, fsu_fault /* Load the fault handler */ mov w5, #1 @@ -73,12 +73,32 @@ ENTRY(casueword32) str w4, [x2] /* Store the read data */ mov w0, w5 /* Result same as store status */ ret /* Return */ -END(casueword32) +END(casueword32_llsc) /* - * int casueword(volatile u_long *, u_long, u_long *, u_long) + * int casueword32_lse(volatile uint32_t *, uint32_t, uint32_t *, uint32_t) */ -ENTRY(casueword) +ENTRY(casueword32_lse) + check_user_access 0, (VM_MAXUSER_ADDRESS-3), fsu_fault_nopcb + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x4) /* And set it */ + ENTER_USER_ACCESS(w6, x4) + mov w7, w1 /* Back up the compare value */ + .arch_extension lse + cas w1, w3, [x0] /* Compare and Swap */ + .arch_extension nolse + cmp w1, w7 /* Check if successful */ + cset w0, ne /* Return 0 on success, 1 on failure */ + EXIT_USER_ACCESS(w6) + SET_FAULT_HANDLER(xzr, x6) /* Reset the fault handler */ + str w1, [x2] /* Store the read data */ + ret /* Return */ +END(casueword32_lse) + +/* + * int casueword_llsc(volatile u_long *, u_long, u_long *, u_long) + */ +ENTRY(casueword_llsc) check_user_access 0, (VM_MAXUSER_ADDRESS-7), fsu_fault_nopcb adr x6, fsu_fault /* Load the fault handler */ mov w5, #1 @@ -93,7 +113,27 @@ ENTRY(casueword) str x4, [x2] /* Store the read data */ mov w0, w5 /* Result same as store status */ ret /* Return */ -END(casueword) +END(casueword_llsc) + +/* + * int casueword_lse(volatile u_long *, u_long, u_long *, u_long) + */ +ENTRY(casueword_lse) + check_user_access 0, (VM_MAXUSER_ADDRESS-3), fsu_fault_nopcb + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x4) /* And set it */ + ENTER_USER_ACCESS(w6, x4) + mov x7, x1 /* Back up the compare value */ + .arch_extension lse + cas x1, x3, [x0] /* Compare and Swap */ + .arch_extension nolse + cmp x1, x7 /* Check if successful */ + cset w0, ne /* Return 0 on success, 1 on failure */ + EXIT_USER_ACCESS(w6) + SET_FAULT_HANDLER(xzr, x6) /* Reset the fault handler */ + str x1, [x2] /* Store the read data */ + ret /* Return */ +END(casueword_lse) .macro fsudata insn, ret_reg, user_arg adr x7, fsu_fault /* Load the fault handler */ diff --git a/sys/arm64/arm64/support_ifunc.c b/sys/arm64/arm64/support_ifunc.c new file mode 100644 index 000000000000..8abf3f3c75af --- /dev/null +++ b/sys/arm64/arm64/support_ifunc.c @@ -0,0 +1,58 @@ +/*- + * Copyright (c) 2022 The FreeBSD Foundation + * + * This software was developed by Andrew Turner under sponsorship from + * the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include + +#include + +#include +#include + +int casueword32_llsc(volatile uint32_t *, uint32_t, uint32_t *, uint32_t); +int casueword32_lse(volatile uint32_t *, uint32_t, uint32_t *, uint32_t); + +int casueword_llsc(volatile u_long *, u_long, u_long *, u_long); +int casueword_lse(volatile u_long *, u_long, u_long *, u_long); + +DEFINE_IFUNC(, int, casueword32, (volatile uint32_t *base, uint32_t oldval, + uint32_t *oldvalp, uint32_t newval)) +{ + if (lse_supported) + return (casueword32_lse); + + return (casueword32_llsc); +} + +DEFINE_IFUNC(, int, casueword, (volatile u_long *base, u_long oldval, + u_long *oldvalp, u_long newval)) +{ + if (lse_supported) + return (casueword_lse); + + return (casueword_llsc); +} diff --git a/sys/conf/files.arm64 b/sys/conf/files.arm64 index 061e566efa70..4bcc4ef122cd 100644 --- a/sys/conf/files.arm64 +++ b/sys/conf/files.arm64 @@ -70,6 +70,7 @@ arm64/arm64/pmap.c standard arm64/arm64/ptrace_machdep.c standard arm64/arm64/sigtramp.S standard arm64/arm64/stack_machdep.c optional ddb | stack +arm64/arm64/support_ifunc.c standard arm64/arm64/support.S standard arm64/arm64/swtch.S standard arm64/arm64/sys_machdep.c standard