From nobody Wed Feb 16 03:04:04 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7DA4019D9874; Wed, 16 Feb 2022 03:04:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jz2q82zpyz4trh; Wed, 16 Feb 2022 03:04:04 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644980644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UG81GXYDD2vw8g1+ENi8VdhiWlLnOCZ3aeeiveb6A7A=; b=NRHN6NrBSw9z/I9saQZ6zaYmG+vJ6FbEz9U1/7SsYRWE/8DNKeMhyUnvj0Izv76oP1CXKH ysvPph+mNiYwy8Yn/dtQFWnk944n8Je6V0a2leHLBPT7+2zy9WWvqvB4i4SBnOijytWhG+ 8njv5ln19sAuka9jRDuQ2NGnPgiutcPMNxmCgnNaSZrUr/Gq4XiSmdjVhWQt2+dWkK6+Js QKPn1umNYMEV1GqAr4CcuxGNtn0p4ULq+LdHO7k6PDhYOhWwo2QWIZlbGjzdkeWT+UlcI7 H5lnaG0PE7OJuDJQVWvKh9A0QMqlPOe9tRkyVvmY+Sbc+k8l1ZKX+XlA2UVbhw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3F2F5673C; Wed, 16 Feb 2022 03:04:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 21G344g1059653; Wed, 16 Feb 2022 03:04:04 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 21G344tW059652; Wed, 16 Feb 2022 03:04:04 GMT (envelope-from git) Date: Wed, 16 Feb 2022 03:04:04 GMT Message-Id: <202202160304.21G344tW059652@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 0b3235ef743d - main - armv8crypto: Factor out some duplicated GCM code List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 0b3235ef743d1561c57989042b3c364a5a955f4f Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644980644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UG81GXYDD2vw8g1+ENi8VdhiWlLnOCZ3aeeiveb6A7A=; b=tTm4keGs6epefS/lRpuAk05vwVFPwU1t4exVuNqBP9gDFBSkzyWAI/UFjkw4LfkxGbf7+M /QBRRGykGvtqQJckWW8vZDD/XdIZvNX0DuWS0B7CiP9AOeM0PU1cLAhRwx972SPgM3Q4YV 08RYSli4Ghcg71+6QDm7LMGbwulA7f98+vI/sjUYImT6KDhAaPKY8Hg70Ltd5PMNa7gA2M HmKIRf/AumSJT1caSAAau5tm7Idn8DfV2vlmqrrElbripLs7oMH4KgdEZmRUp0Cxa8g+X/ obOAdkHB1RL/6JZwa7J+yBTlNhQZ+3FEXi5avKpw+ObAsXd9asF2DfdMWUD7Jw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644980644; a=rsa-sha256; cv=none; b=ABJz4Mu4YNa3+y4HdZt6BokvVJ1/mqyyTAE+0Xw2QU5cpN6A1b11VOTaLIgd+jC2nmv8Z4 bpK00N6HZDwPio81Q2WdqYml/nul5zbEveK2GeOQhimoAMPZtlgqS43NZ/2llV//2G2I5k hSqswuXbiiCEH8+brgVWQhEu7HTC73hUMfdl6Z+W5zi+gNvbFaS8vUP16f1B7ThOxPn0J0 JAxqh8joHRl66TCxsla+r+6rbre88zTGWFTq9SrEcQsnG8C67sEYwYnyvQ2V/zvhQ+FcIy epB1z/mqpLPP6Cb4oGI+DBcIre68h0xOp1ZNHW+FC69jtok9TndlZszRuKhGMg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=0b3235ef743d1561c57989042b3c364a5a955f4f commit 0b3235ef743d1561c57989042b3c364a5a955f4f Author: Mark Johnston AuthorDate: 2022-02-16 02:45:59 +0000 Commit: Mark Johnston CommitDate: 2022-02-16 02:47:41 +0000 armv8crypto: Factor out some duplicated GCM code This is in preparation for using buffer cursors. No functional change intended. Reviewed by: jhb Sponsored by: Ampere Computing LLC Submitted by: Klara Inc. MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D28948 --- sys/crypto/armv8/armv8_crypto_wrap.c | 111 ++++++++++++++++------------------- 1 file changed, 51 insertions(+), 60 deletions(-) diff --git a/sys/crypto/armv8/armv8_crypto_wrap.c b/sys/crypto/armv8/armv8_crypto_wrap.c index 3c0223964ee4..b5aee0cc1cf6 100644 --- a/sys/crypto/armv8/armv8_crypto_wrap.c +++ b/sys/crypto/armv8/armv8_crypto_wrap.c @@ -249,46 +249,71 @@ struct armv8_gcm_state { uint8_t aes_counter[AES_BLOCK_LEN]; }; -void -armv8_aes_encrypt_gcm(AES_key_t *aes_key, size_t len, - const uint8_t *from, uint8_t *to, - size_t authdatalen, const uint8_t *authdata, - uint8_t tag[static GMAC_DIGEST_LEN], - const uint8_t iv[static AES_GCM_IV_LEN], - const __uint128_val_t *Htable) +static void +armv8_aes_gmac_setup(struct armv8_gcm_state *s, AES_key_t *aes_key, + const uint8_t *authdata, size_t authdatalen, + const uint8_t iv[static AES_GCM_IV_LEN], const __uint128_val_t *Htable) { - struct armv8_gcm_state s; - const uint64_t *from64; - uint64_t *to64; uint8_t block[AES_BLOCK_LEN]; - size_t i, trailer; + size_t trailer; - bzero(&s.aes_counter, AES_BLOCK_LEN); - memcpy(s.aes_counter, iv, AES_GCM_IV_LEN); + bzero(s->aes_counter, AES_BLOCK_LEN); + memcpy(s->aes_counter, iv, AES_GCM_IV_LEN); /* Setup the counter */ - s.aes_counter[AES_BLOCK_LEN - 1] = 1; + s->aes_counter[AES_BLOCK_LEN - 1] = 1; /* EK0 for a final GMAC round */ - aes_v8_encrypt(s.aes_counter, s.EK0.c, aes_key); + aes_v8_encrypt(s->aes_counter, s->EK0.c, aes_key); /* GCM starts with 2 as counter, 1 is used for final xor of tag. */ - s.aes_counter[AES_BLOCK_LEN - 1] = 2; + s->aes_counter[AES_BLOCK_LEN - 1] = 2; - memset(s.Xi.c, 0, sizeof(s.Xi.c)); + memset(s->Xi.c, 0, sizeof(s->Xi.c)); trailer = authdatalen % AES_BLOCK_LEN; if (authdatalen - trailer > 0) { - gcm_ghash_v8(s.Xi.u, Htable, authdata, authdatalen - trailer); + gcm_ghash_v8(s->Xi.u, Htable, authdata, authdatalen - trailer); authdata += authdatalen - trailer; } if (trailer > 0 || authdatalen == 0) { memset(block, 0, sizeof(block)); memcpy(block, authdata, trailer); - gcm_ghash_v8(s.Xi.u, Htable, block, AES_BLOCK_LEN); + gcm_ghash_v8(s->Xi.u, Htable, block, AES_BLOCK_LEN); } +} - from64 = (const uint64_t*)from; - to64 = (uint64_t*)to; +static void +armv8_aes_gmac_finish(struct armv8_gcm_state *s, size_t len, + size_t authdatalen, const __uint128_val_t *Htable) +{ + /* Lengths block */ + s->lenblock.u[0] = s->lenblock.u[1] = 0; + s->lenblock.d[1] = htobe32(authdatalen * 8); + s->lenblock.d[3] = htobe32(len * 8); + gcm_ghash_v8(s->Xi.u, Htable, s->lenblock.c, AES_BLOCK_LEN); + + s->Xi.u[0] ^= s->EK0.u[0]; + s->Xi.u[1] ^= s->EK0.u[1]; +} + +void +armv8_aes_encrypt_gcm(AES_key_t *aes_key, size_t len, + const uint8_t *from, uint8_t *to, + size_t authdatalen, const uint8_t *authdata, + uint8_t tag[static GMAC_DIGEST_LEN], + const uint8_t iv[static AES_GCM_IV_LEN], + const __uint128_val_t *Htable) +{ + struct armv8_gcm_state s; + const uint64_t *from64; + uint64_t *to64; + uint8_t block[AES_BLOCK_LEN]; + size_t i, trailer; + + armv8_aes_gmac_setup(&s, aes_key, authdata, authdatalen, iv, Htable); + + from64 = (const uint64_t *)from; + to64 = (uint64_t *)to; trailer = len % AES_BLOCK_LEN; for (i = 0; i < (len - trailer); i += AES_BLOCK_LEN) { @@ -316,14 +341,7 @@ armv8_aes_encrypt_gcm(AES_key_t *aes_key, size_t len, gcm_ghash_v8(s.Xi.u, Htable, block, AES_BLOCK_LEN); } - /* Lengths block */ - s.lenblock.u[0] = s.lenblock.u[1] = 0; - s.lenblock.d[1] = htobe32(authdatalen * 8); - s.lenblock.d[3] = htobe32(len * 8); - gcm_ghash_v8(s.Xi.u, Htable, s.lenblock.c, AES_BLOCK_LEN); - - s.Xi.u[0] ^= s.EK0.u[0]; - s.Xi.u[1] ^= s.EK0.u[1]; + armv8_aes_gmac_finish(&s, len, authdatalen, Htable); memcpy(tag, s.Xi.c, GMAC_DIGEST_LEN); explicit_bzero(&s, sizeof(s)); @@ -345,26 +363,8 @@ armv8_aes_decrypt_gcm(AES_key_t *aes_key, size_t len, int error; error = 0; - bzero(&s.aes_counter, AES_BLOCK_LEN); - memcpy(s.aes_counter, iv, AES_GCM_IV_LEN); - - /* Setup the counter */ - s.aes_counter[AES_BLOCK_LEN - 1] = 1; - - /* EK0 for a final GMAC round */ - aes_v8_encrypt(s.aes_counter, s.EK0.c, aes_key); - memset(s.Xi.c, 0, sizeof(s.Xi.c)); - trailer = authdatalen % AES_BLOCK_LEN; - if (authdatalen - trailer > 0) { - gcm_ghash_v8(s.Xi.u, Htable, authdata, authdatalen - trailer); - authdata += authdatalen - trailer; - } - if (trailer > 0 || authdatalen == 0) { - memset(block, 0, sizeof(block)); - memcpy(block, authdata, trailer); - gcm_ghash_v8(s.Xi.u, Htable, block, AES_BLOCK_LEN); - } + armv8_aes_gmac_setup(&s, aes_key, authdata, authdatalen, iv, Htable); trailer = len % AES_BLOCK_LEN; if (len - trailer > 0) @@ -375,24 +375,15 @@ armv8_aes_decrypt_gcm(AES_key_t *aes_key, size_t len, gcm_ghash_v8(s.Xi.u, Htable, block, AES_BLOCK_LEN); } - /* Lengths block */ - s.lenblock.u[0] = s.lenblock.u[1] = 0; - s.lenblock.d[1] = htobe32(authdatalen * 8); - s.lenblock.d[3] = htobe32(len * 8); - gcm_ghash_v8(s.Xi.u, Htable, s.lenblock.c, AES_BLOCK_LEN); + armv8_aes_gmac_finish(&s, len, authdatalen, Htable); - s.Xi.u[0] ^= s.EK0.u[0]; - s.Xi.u[1] ^= s.EK0.u[1]; if (timingsafe_bcmp(tag, s.Xi.c, GMAC_DIGEST_LEN) != 0) { error = EBADMSG; goto out; } - /* GCM starts with 2 as counter, 1 is used for final xor of tag. */ - s.aes_counter[AES_BLOCK_LEN - 1] = 2; - - from64 = (const uint64_t*)from; - to64 = (uint64_t*)to; + from64 = (const uint64_t *)from; + to64 = (uint64_t *)to; for (i = 0; i < (len - trailer); i += AES_BLOCK_LEN) { aes_v8_encrypt(s.aes_counter, s.EKi.c, aes_key);