From nobody Fri Mar 29 18:11:46 2024 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V5pQB3rmHz5FN8t; Fri, 29 Mar 2024 18:11:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V5pQB2cFmz4h2t; Fri, 29 Mar 2024 18:11:46 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711735906; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=xRne4va5ewm9I0IlEGgWDe5md6y5j/YNK27JKXkRsN0=; b=XOqcrehpwawj+6Ai2DXLPUX5K6vOtwrGg4d3XpOG8pCpkBWVgxPZLfUfe9sYANGvcTNSnc 5zOQKX6wEMQnCjly1Fg2ziSVCet6z3gbpslgQM+AY7zneh2RNS458sOyQPl/vYkK28UcPu mYZfFOnFZkAE0Io8qNFypeKd1An2IR5O9b1cDB+J8346tHUu+sf0cOAzpAC9/GWWuZRxz/ 9hp6uQSuR47CvgIulSici+umRXpsT7Kq4+R0CAOGnKQUfUeocW1klwx/RDosCLALAfqEus D0JIHqn/zEhPuygifex+suRR0sZjBTKdhowQQ5/78QVvG2/aE553JgTH6TbHAQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711735906; a=rsa-sha256; cv=none; b=EBWxX/hQaBlAckneUl0Zevcb2K7cMjPyl7aS7vfEuBx7zYrWR8DE8fcfK3qYXTXg6mR/8z gi3IZvZ3enm1H5EhHQK+YcJ2WQexv2u9rfyxodcyqBawCDWQyndBenjLObNk6ruaD3mJCo euWM8du/1u4U7LtrNTXa6ZP+DOzQ7hY5I99x5YQze3VtUzHvR+8YJZN452j12v8M6k0BCG gdtAIzRytXcew3zq4R1MlsFZqbUNX6iIHc7I2f76ZfbhfPKw1hthmIP1kG/j61VE0hhxaN Pm7vXTghf1Xly9yX69JPD2s9a+FFE+k0URiAwuA812wgekjAGGEFArho4uzNfg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711735906; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=xRne4va5ewm9I0IlEGgWDe5md6y5j/YNK27JKXkRsN0=; b=UMU2CdFY1GSBe+zn6rJ+7rGB3HhrwCJTnk5wosaRguYKp7NARvNoW/1HNqdW05KJt7atya Imrhl/dm0aSoYb/Rfva9/2BVRPmGS1XWl6S5uHQFg7/ClepHo7Fn2JKPZ3/Nw8trsxsi/F ydVsDQ5UzyL6HNO371b6YEWyTMcpv6LP6KofWPJANiG+RKhQO8kFMUP9bF29tyTEi9SlZa LYIVnifwWLLUou6zcUFWLGdTBwLWuI+aGODrNv3tgbzDYWKj99slr5kLBEfg1kzdNXnHN3 SV0plpV1+/iLojFEl9oc8zPAAVh1Ww/Z1qnJbBZamM/RRS803sxhZm/6aEUwpA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V5pQB222szLyy; Fri, 29 Mar 2024 18:11:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 42TIBkRA078032; Fri, 29 Mar 2024 18:11:46 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 42TIBk88078029; Fri, 29 Mar 2024 18:11:46 GMT (envelope-from git) Date: Fri, 29 Mar 2024 18:11:46 GMT Message-Id: <202403291811.42TIBk88078029@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 15e12749ef1d - stable/14 - ossl: Add support for armv7 List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 15e12749ef1dd55a18a7b1e5672e2b75f1bde68b Auto-Submitted: auto-generated The branch stable/14 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=15e12749ef1dd55a18a7b1e5672e2b75f1bde68b commit 15e12749ef1dd55a18a7b1e5672e2b75f1bde68b Author: Mark Johnston AuthorDate: 2023-11-30 17:46:08 +0000 Commit: Mark Johnston CommitDate: 2024-03-29 13:53:05 +0000 ossl: Add support for armv7 OpenSSL provides implementations of several AES modes which use bitslicing and can be accelerated on CPUs which support the NEON extension. This patch adds arm platform support to ossl(4) and provides an AES-CBC implementation, though bsaes_cbc_encrypt() only implements decryption. The real goal is to provide an accelerated AES-GCM implementation; this will be added in a subsequent patch. Initially derived from https://reviews.freebsd.org/D37420. Reviewed by: jhb Sponsored by: Klara, Inc. Sponsored by: Stormshield MFC after: 3 months Differential Revision: https://reviews.freebsd.org/D41304 (cherry picked from commit 44f8e1e8530e1d2e95e84bbbe3d22ac9cb2557fe) --- sys/conf/files.arm | 10 +++++ sys/crypto/openssl/arm/arm_arch.h | 84 ++++++++++++++++++++++++++++++++++ sys/crypto/openssl/ossl_aes.c | 2 + sys/crypto/openssl/ossl_aes_gcm.h | 6 +-- sys/crypto/openssl/ossl_arm.c | 59 ++++++++++++++++++++++++ sys/crypto/openssl/ossl_arm.h | 94 +++++++++++++++++++++++++++++++++++++++ sys/crypto/openssl/ossl_cipher.h | 8 ++++ sys/modules/Makefile | 6 ++- sys/modules/ossl/Makefile | 12 +++++ 9 files changed, 275 insertions(+), 6 deletions(-) diff --git a/sys/conf/files.arm b/sys/conf/files.arm index 94a8755fd315..3800f4345765 100644 --- a/sys/conf/files.arm +++ b/sys/conf/files.arm @@ -133,6 +133,16 @@ libkern/ucmpdi2.c standard libkern/udivdi3.c standard libkern/umoddi3.c standard +crypto/openssl/ossl_arm.c optional ossl +crypto/openssl/arm/aes-armv4.S optional ossl +crypto/openssl/arm/bsaes-armv7.S optional ossl \ + compile-with "${CC} -D__KERNEL__ -c ${CFLAGS:N-mgeneral-regs-only} ${WERROR} ${.IMPSRC}" +crypto/openssl/arm/chacha-armv4.S optional ossl +crypto/openssl/arm/poly1305-armv4.S optional ossl +crypto/openssl/arm/sha1-armv4-large.S optional ossl +crypto/openssl/arm/sha256-armv4.S optional ossl +crypto/openssl/arm/sha512-armv4.S optional ossl + # Annapurna support arm/annapurna/alpine/alpine_ccu.c optional al_ccu fdt arm/annapurna/alpine/alpine_nb_service.c optional al_nb_service fdt diff --git a/sys/crypto/openssl/arm/arm_arch.h b/sys/crypto/openssl/arm/arm_arch.h new file mode 100644 index 000000000000..8b7105571d78 --- /dev/null +++ b/sys/crypto/openssl/arm/arm_arch.h @@ -0,0 +1,84 @@ +/* + * Copyright 2011-2018 The OpenSSL Project Authors. All Rights Reserved. + * + * Licensed under the OpenSSL license (the "License"). You may not use + * this file except in compliance with the License. You can obtain a copy + * in the file LICENSE in the source distribution or at + * https://www.openssl.org/source/license.html + */ + +#ifndef OSSL_CRYPTO_ARM_ARCH_H +# define OSSL_CRYPTO_ARM_ARCH_H + +# if !defined(__ARM_ARCH__) +# if defined(__CC_ARM) +# define __ARM_ARCH__ __TARGET_ARCH_ARM +# if defined(__BIG_ENDIAN) +# define __ARMEB__ +# else +# define __ARMEL__ +# endif +# elif defined(__GNUC__) +# if defined(__aarch64__) +# define __ARM_ARCH__ 8 +# if __BYTE_ORDER__==__ORDER_BIG_ENDIAN__ +# define __ARMEB__ +# else +# define __ARMEL__ +# endif + /* + * Why doesn't gcc define __ARM_ARCH__? Instead it defines + * bunch of below macros. See all_architectures[] table in + * gcc/config/arm/arm.c. On a side note it defines + * __ARMEL__/__ARMEB__ for little-/big-endian. + */ +# elif defined(__ARM_ARCH) +# define __ARM_ARCH__ __ARM_ARCH +# elif defined(__ARM_ARCH_8A__) +# define __ARM_ARCH__ 8 +# elif defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) || \ + defined(__ARM_ARCH_7R__)|| defined(__ARM_ARCH_7M__) || \ + defined(__ARM_ARCH_7EM__) +# define __ARM_ARCH__ 7 +# elif defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) || \ + defined(__ARM_ARCH_6K__)|| defined(__ARM_ARCH_6M__) || \ + defined(__ARM_ARCH_6Z__)|| defined(__ARM_ARCH_6ZK__) || \ + defined(__ARM_ARCH_6T2__) +# define __ARM_ARCH__ 6 +# elif defined(__ARM_ARCH_5__) || defined(__ARM_ARCH_5T__) || \ + defined(__ARM_ARCH_5E__)|| defined(__ARM_ARCH_5TE__) || \ + defined(__ARM_ARCH_5TEJ__) +# define __ARM_ARCH__ 5 +# elif defined(__ARM_ARCH_4__) || defined(__ARM_ARCH_4T__) +# define __ARM_ARCH__ 4 +# else +# error "unsupported ARM architecture" +# endif +# endif +# endif + +# if !defined(__ARM_MAX_ARCH__) +# define __ARM_MAX_ARCH__ __ARM_ARCH__ +# endif + +# if __ARM_MAX_ARCH__<__ARM_ARCH__ +# error "__ARM_MAX_ARCH__ can't be less than __ARM_ARCH__" +# elif __ARM_MAX_ARCH__!=__ARM_ARCH__ +# if __ARM_ARCH__<7 && __ARM_MAX_ARCH__>=7 && defined(__ARMEB__) +# error "can't build universal big-endian binary" +# endif +# endif + +# ifndef __ASSEMBLER__ +extern unsigned int OPENSSL_armcap_P; +# endif + +# define ARMV7_NEON (1<<0) +# define ARMV7_TICK (1<<1) +# define ARMV8_AES (1<<2) +# define ARMV8_SHA1 (1<<3) +# define ARMV8_SHA256 (1<<4) +# define ARMV8_PMULL (1<<5) +# define ARMV8_SHA512 (1<<6) + +#endif diff --git a/sys/crypto/openssl/ossl_aes.c b/sys/crypto/openssl/ossl_aes.c index 800518e51205..9560a8149a7c 100644 --- a/sys/crypto/openssl/ossl_aes.c +++ b/sys/crypto/openssl/ossl_aes.c @@ -40,6 +40,8 @@ #include #elif defined (__aarch64__) #include +#elif defined (__arm__) +#include #endif static ossl_cipher_process_t ossl_aes_cbc; diff --git a/sys/crypto/openssl/ossl_aes_gcm.h b/sys/crypto/openssl/ossl_aes_gcm.h index cd0c1e324354..90511318da6c 100644 --- a/sys/crypto/openssl/ossl_aes_gcm.h +++ b/sys/crypto/openssl/ossl_aes_gcm.h @@ -28,7 +28,6 @@ #define _OSSL_AES_GCM_H_ #include -#include struct ossl_gcm_context; @@ -64,10 +63,7 @@ struct ossl_gcm_context { unsigned int mres, ares; } gcm; - struct { - uint32_t ks[4 * (RIJNDAEL_MAXNR + 1)]; - int rounds; - } aes_ks; + struct ossl_aes_keysched aes_ks; const struct ossl_aes_gcm_ops *ops; }; diff --git a/sys/crypto/openssl/ossl_arm.c b/sys/crypto/openssl/ossl_arm.c new file mode 100644 index 000000000000..1ec95acd74cd --- /dev/null +++ b/sys/crypto/openssl/ossl_arm.c @@ -0,0 +1,59 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause-FreeBSD + * + * Copyright (c) 2023 Stormshield + * Copyright (c) 2023 Semihalf + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer, + * without modification. + * 2. Redistributions in binary form must reproduce at minimum a disclaimer + * similar to the "NO WARRANTY" disclaimer below ("Disclaimer") and any + * redistribution must be conditioned upon including a substantially + * similar Disclaimer requirement for further binary redistribution. + * + * NO WARRANTY + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTIBILITY + * AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, + * OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER + * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF + * THE POSSIBILITY OF SUCH DAMAGES. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include + +#include +#include + +#include +#include +#include + +ossl_cipher_setkey_t AES_set_encrypt_key; +ossl_cipher_setkey_t AES_set_decrypt_key; + +unsigned int OPENSSL_armcap_P; + +void +ossl_cpuid(struct ossl_softc *sc) +{ + if (elf_hwcap & HWCAP_NEON) { + OPENSSL_armcap_P |= ARMV7_NEON; + + sc->has_aes = true; + ossl_cipher_aes_cbc.set_encrypt_key = AES_set_encrypt_key; + ossl_cipher_aes_cbc.set_decrypt_key = AES_set_decrypt_key; + } +} diff --git a/sys/crypto/openssl/ossl_arm.h b/sys/crypto/openssl/ossl_arm.h new file mode 100644 index 000000000000..56772feb43f2 --- /dev/null +++ b/sys/crypto/openssl/ossl_arm.h @@ -0,0 +1,94 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause-FreeBSD + * + * Copyright (c) 2023 Stormshield + * Copyright (c) 2023 Semihalf + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer, + * without modification. + * 2. Redistributions in binary form must reproduce at minimum a disclaimer + * similar to the "NO WARRANTY" disclaimer below ("Disclaimer") and any + * redistribution must be conditioned upon including a substantially + * similar Disclaimer requirement for further binary redistribution. + * + * NO WARRANTY + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTIBILITY + * AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, + * OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER + * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF + * THE POSSIBILITY OF SUCH DAMAGES. + */ + +#ifndef __OSSL_ARM__ +#define __OSSL_ARM__ + +#include +#include + +#include + +struct bsaes_key { + struct ossl_aes_keysched ks; + int converted; +#define BSAES_KEY_SIZE (128 * (RIJNDAEL_MAXNR - 1) + 2 * AES_BLOCK_LEN) + uint8_t bitslice[BSAES_KEY_SIZE] __aligned(8); +} __aligned(8); + +ossl_cipher_encrypt_t ossl_bsaes_cbc_encrypt; + +void AES_encrypt(const void *, void *, const void *); + +static inline void +AES_CBC_ENCRYPT(const unsigned char *in, unsigned char *out, + size_t length, const void *key, unsigned char *iv, int encrypt) +{ + struct bsaes_key bsks; + uint32_t iv32[4], scratch[4]; + + /* + * bsaes_cbc_encrypt has some special requirements w.r.t input data. + * The key buffer, that normally holds round keys is used as a scratch + * space. 128 bytes per round of extra space is required. + * Another thing is that only decryption is supported. + * In the case of encryption block chaining has to be done in C. + */ + if (!encrypt) { + memcpy(&bsks.ks, key, sizeof(bsks.ks)); + bsks.converted = 0; + ossl_bsaes_cbc_encrypt(in, out, length, &bsks, iv, false); + return; + } + + length /= AES_BLOCK_LEN; + memcpy(iv32, iv, AES_BLOCK_LEN); + + while (length-- > 0) { + memcpy(scratch, in, AES_BLOCK_LEN); + + /* XOR plaintext with IV. */ + scratch[0] ^= iv32[0]; + scratch[1] ^= iv32[1]; + scratch[2] ^= iv32[2]; + scratch[3] ^= iv32[3]; + + AES_encrypt(scratch, out, key); + + memcpy(iv32, out, AES_BLOCK_LEN); + in += AES_BLOCK_LEN; + out += AES_BLOCK_LEN; + } + + memcpy(iv, iv32, AES_BLOCK_LEN); +} + +#endif /* __OSSL_ARM__ */ diff --git a/sys/crypto/openssl/ossl_cipher.h b/sys/crypto/openssl/ossl_cipher.h index 6599524a707f..886fd9f83350 100644 --- a/sys/crypto/openssl/ossl_cipher.h +++ b/sys/crypto/openssl/ossl_cipher.h @@ -28,6 +28,9 @@ #ifndef __OSSL_CIPHER_H__ #define __OSSL_CIPHER_H__ +#include +#include + struct ossl_session_cipher; struct cryptop; struct crypto_session_params; @@ -50,4 +53,9 @@ struct ossl_cipher { ossl_cipher_process_t *process; }; +struct ossl_aes_keysched { + uint32_t ks[4 * (RIJNDAEL_MAXNR + 1)]; + int rounds; +}; + #endif diff --git a/sys/modules/Makefile b/sys/modules/Makefile index b34d20dd9147..4dedd440017f 100644 --- a/sys/modules/Makefile +++ b/sys/modules/Makefile @@ -565,7 +565,6 @@ _iwlwifi= iwlwifi .if ${MK_SOURCELESS_UCODE} != "no" _iwlwififw= iwlwififw .endif -_ossl= ossl _rtw88= rtw88 .if ${MK_SOURCELESS_UCODE} != "no" _rtw88fw= rtw88fw @@ -573,6 +572,11 @@ _rtw88fw= rtw88fw _vmware= vmware .endif +.if ${MACHINE_CPUARCH} == "aarch64" || ${MACHINE_CPUARCH} == "amd64" || \ + ${MACHINE_CPUARCH} == "i386" || ${MACHINE_ARCH} == "armv7" +_ossl= ossl +.endif + # MAC framework .if ${KERN_OPTS:MMAC} || defined(ALL_MODULES) _mac_biba= mac_biba diff --git a/sys/modules/ossl/Makefile b/sys/modules/ossl/Makefile index 7f70f19db76c..d26aabf7bff2 100644 --- a/sys/modules/ossl/Makefile +++ b/sys/modules/ossl/Makefile @@ -16,6 +16,16 @@ SRCS= bus_if.h \ ossl_sha512.c \ ${SRCS.${MACHINE_CPUARCH}} +SRCS.arm= \ + aes-armv4.S \ + bsaes-armv7.S \ + chacha-armv4.S \ + poly1305-armv4.S \ + sha1-armv4-large.S \ + sha256-armv4.S \ + sha512-armv4.S \ + ossl_arm.c + SRCS.aarch64= \ chacha-armv8.S \ poly1305-armv8.S \ @@ -47,6 +57,8 @@ SRCS.i386= \ sha512-586.S \ ossl_x86.c +CFLAGS.bsaes-armv7.S+= -D__KERNEL__ + # For arm64, we are forced to rewrite the compiler invocation for the assembly # files, to remove -mgeneral-regs-only. ${SRCS.aarch64:M*.S:S/S/o/}: ${.TARGET:R}.S