From nobody Tue Jan 11 22:38:13 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8CE16194C3F6; Tue, 11 Jan 2022 22:38:16 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JYQZb0hf5z4mDL; Tue, 11 Jan 2022 22:38:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0A6CD21A33; Tue, 11 Jan 2022 22:38:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 20BMcDVE075930; Tue, 11 Jan 2022 22:38:13 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 20BMcDqR075929; Tue, 11 Jan 2022 22:38:13 GMT (envelope-from git) Date: Tue, 11 Jan 2022 22:38:13 GMT Message-Id: <202201112238.20BMcDqR075929@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: a221a8f4a0de - main - cryptosoft: Use multi-block encrypt/decrypt for AES-GCM. List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: a221a8f4a0deb6619448e9d2df696625d454ab3d Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641940696; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tQ2mq5F0G0oL8uMKKbhGQqIRXRUGO9ijgqhdRt6EKI4=; b=NajsTpp/5El3avK7vXOomaLpZTiDuoOUmwKt9L7ymOoyrdVb21rlMpFdpulaLqP/zPU7fF 2WWeoRCbKnuP1ouJiHbadGaD35WVkHnqx4aTJBAw/Vv2lwiR+DeeEopDW7qzR+w2YJFSpO qyJuTJdIwNGFl1Pqn1ERhjVXs2yapU8sVnIHcvUHvGvu+M9TgzEigzDDkhamhD3czWVfYm QJ2YfjyOLGbJdDuaAZVRp0GYbhKNuSdYrl139YMQXMLbQUVcQjaXU7UYxuIFJT8NfMK0cS u0GHKGxUmYMqzWCVf1ZUqoQS9vhTRPMi7/OxmGbthA0KysVBEDoC0E4DFoCpgA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1641940696; a=rsa-sha256; cv=none; b=pkfbCIzfewZ9tjpqmayl7FdfbuR84DJUU6LpOL9JfqfBtwjoRIWDMJkqHzaDh57lbambj4 bTL1ASqspghMvyvKLEDYI8QNwmMw8OOjdvIgHZUa/xht9lDPbSaK+T1vxXlggecI9DiXKz aqNQBnhcKYHvyJurpXX2e6Drz5wjVuRpLVRokMtHhS4XTZTCOQCnduYuwQy17tHE8G9VXx t2Gg4etmRdNl+uOXDloRZyw0aZBaCl3fDJ70oNp/blFKbnIWw1u+WuGQZvMre4TNi/u9D4 AewBdzz2iiAbvSMD1cR/TULr6aFKwl2O0aP7XulEK7a6s9tOoDkwtHk+MWbAEg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=a221a8f4a0deb6619448e9d2df696625d454ab3d commit a221a8f4a0deb6619448e9d2df696625d454ab3d Author: John Baldwin AuthorDate: 2022-01-11 22:20:32 +0000 Commit: John Baldwin CommitDate: 2022-01-11 22:20:32 +0000 cryptosoft: Use multi-block encrypt/decrypt for AES-GCM. Reviewed by: markj Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D33756 --- sys/opencrypto/cryptosoft.c | 130 +++++++++++++++++++++++++++++++------------- 1 file changed, 91 insertions(+), 39 deletions(-) diff --git a/sys/opencrypto/cryptosoft.c b/sys/opencrypto/cryptosoft.c index 864873944fd4..36081589cc65 100644 --- a/sys/opencrypto/cryptosoft.c +++ b/sys/opencrypto/cryptosoft.c @@ -411,12 +411,12 @@ swcr_gcm(const struct swcr_session *ses, struct cryptop *crp) struct crypto_buffer_cursor cc_in, cc_out; const u_char *inblk; u_char *outblk; + size_t inlen, outlen, todo; const struct swcr_auth *swa; const struct swcr_encdec *swe; const struct enc_xform *exf; void *ctx; uint32_t *blkp; - size_t len; int blksz, error, ivlen, r, resid; swa = &ses->swcr_auth; @@ -441,30 +441,30 @@ swcr_gcm(const struct swcr_session *ses, struct cryptop *crp) /* Supply MAC with AAD */ if (crp->crp_aad != NULL) { - len = rounddown(crp->crp_aad_length, blksz); - if (len != 0) - exf->update(ctx, crp->crp_aad, len); - if (crp->crp_aad_length != len) { + inlen = rounddown2(crp->crp_aad_length, blksz); + if (inlen != 0) + exf->update(ctx, crp->crp_aad, inlen); + if (crp->crp_aad_length != inlen) { memset(blk, 0, blksz); - memcpy(blk, (char *)crp->crp_aad + len, - crp->crp_aad_length - len); + memcpy(blk, (char *)crp->crp_aad + inlen, + crp->crp_aad_length - inlen); exf->update(ctx, blk, blksz); } } else { crypto_cursor_init(&cc_in, &crp->crp_buf); crypto_cursor_advance(&cc_in, crp->crp_aad_start); for (resid = crp->crp_aad_length; resid >= blksz; - resid -= len) { - inblk = crypto_cursor_segment(&cc_in, &len); - if (len >= blksz) { - len = rounddown(MIN(len, resid), blksz); - crypto_cursor_advance(&cc_in, len); + resid -= inlen) { + inblk = crypto_cursor_segment(&cc_in, &inlen); + if (inlen >= blksz) { + inlen = rounddown2(MIN(inlen, resid), blksz); + crypto_cursor_advance(&cc_in, inlen); } else { - len = blksz; - crypto_cursor_copydata(&cc_in, len, blk); + inlen = blksz; + crypto_cursor_copydata(&cc_in, inlen, blk); inblk = blk; } - exf->update(ctx, inblk, len); + exf->update(ctx, inblk, inlen); } if (resid > 0) { memset(blk, 0, blksz); @@ -476,31 +476,57 @@ swcr_gcm(const struct swcr_session *ses, struct cryptop *crp) /* Do encryption with MAC */ crypto_cursor_init(&cc_in, &crp->crp_buf); crypto_cursor_advance(&cc_in, crp->crp_payload_start); + inblk = crypto_cursor_segment(&cc_in, &inlen); if (CRYPTO_HAS_OUTPUT_BUFFER(crp)) { crypto_cursor_init(&cc_out, &crp->crp_obuf); crypto_cursor_advance(&cc_out, crp->crp_payload_output_start); } else cc_out = cc_in; - for (resid = crp->crp_payload_length; resid >= blksz; resid -= blksz) { - inblk = crypto_cursor_segment(&cc_in, &len); - if (len < blksz) { + outblk = crypto_cursor_segment(&cc_out, &outlen); + + for (resid = crp->crp_payload_length; resid >= blksz; resid -= todo) { + if (inlen < blksz) { crypto_cursor_copydata(&cc_in, blksz, blk); inblk = blk; - } else { - crypto_cursor_advance(&cc_in, blksz); + inlen = blksz; } + if (CRYPTO_OP_IS_ENCRYPT(crp->crp_op)) { - outblk = crypto_cursor_segment(&cc_out, &len); - if (len < blksz) + if (outlen < blksz) { outblk = blk; - exf->encrypt(ctx, inblk, outblk); - exf->update(ctx, outblk, blksz); - if (outblk == blk) + outlen = blksz; + } + + todo = rounddown2(MIN(resid, MIN(inlen, outlen)), + blksz); + + exf->encrypt_multi(ctx, inblk, outblk, todo); + exf->update(ctx, outblk, todo); + + if (outblk == blk) { crypto_cursor_copyback(&cc_out, blksz, blk); - else - crypto_cursor_advance(&cc_out, blksz); + outblk = crypto_cursor_segment(&cc_out, &outlen); + } else { + crypto_cursor_advance(&cc_out, todo); + outlen -= todo; + outblk += todo; + if (outlen == 0) + outblk = crypto_cursor_segment(&cc_out, + &outlen); + } } else { - exf->update(ctx, inblk, blksz); + todo = rounddown2(MIN(resid, inlen), blksz); + exf->update(ctx, inblk, todo); + } + + if (inblk == blk) { + inblk = crypto_cursor_segment(&cc_in, &inlen); + } else { + crypto_cursor_advance(&cc_in, todo); + inlen -= todo; + inblk += todo; + if (inlen == 0) + inblk = crypto_cursor_segment(&cc_in, &inlen); } } if (resid > 0) { @@ -537,22 +563,48 @@ swcr_gcm(const struct swcr_session *ses, struct cryptop *crp) /* tag matches, decrypt data */ crypto_cursor_init(&cc_in, &crp->crp_buf); crypto_cursor_advance(&cc_in, crp->crp_payload_start); + inblk = crypto_cursor_segment(&cc_in, &inlen); + for (resid = crp->crp_payload_length; resid > blksz; - resid -= blksz) { - inblk = crypto_cursor_segment(&cc_in, &len); - if (len < blksz) { + resid -= todo) { + if (inlen < blksz) { crypto_cursor_copydata(&cc_in, blksz, blk); inblk = blk; - } else - crypto_cursor_advance(&cc_in, blksz); - outblk = crypto_cursor_segment(&cc_out, &len); - if (len < blksz) + inlen = blksz; + } + if (outlen < blksz) { outblk = blk; - exf->decrypt(ctx, inblk, outblk); - if (outblk == blk) + outlen = blksz; + } + + todo = rounddown2(MIN(resid, MIN(inlen, outlen)), + blksz); + + exf->decrypt_multi(ctx, inblk, outblk, todo); + + if (inblk == blk) { + inblk = crypto_cursor_segment(&cc_in, &inlen); + } else { + crypto_cursor_advance(&cc_in, todo); + inlen -= todo; + inblk += todo; + if (inlen == 0) + inblk = crypto_cursor_segment(&cc_in, + &inlen); + } + + if (outblk == blk) { crypto_cursor_copyback(&cc_out, blksz, blk); - else - crypto_cursor_advance(&cc_out, blksz); + outblk = crypto_cursor_segment(&cc_out, + &outlen); + } else { + crypto_cursor_advance(&cc_out, todo); + outlen -= todo; + outblk += todo; + if (outlen == 0) + outblk = crypto_cursor_segment(&cc_out, + &outlen); + } } if (resid > 0) { crypto_cursor_copydata(&cc_in, resid, blk);