From nobody Thu Nov 19 15:42:38 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4jVjQk4v68z71GP0 for ; Thu, 19 Nov 2026 15:42:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4jVjQk2yDjz3BxD for ; Thu, 19 Nov 2026 15:42:38 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1795102958; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=qArDrwOZbzNLcRoFyBC9HkL01wvILOuO1lIduePHq78=; b=A4t4ojqgy5p6Lk9L3DpFJlGi3Q4l1PFNYQRR0JDfzyfqDiP/CL3R+BzmBnHn0GNMFb4K82 DCYNq/I3oXef4+S+eXz/s7TU+1Hd3tmXv+a/VINGcY44rywpLCjzX/vyVDJUjk1h6nnoBz N4OC+T/vDwjSkvPfmz3NUxto/A+aystaEiyYocoehUjTTwaRO6O1jqbi9mZKKcjFeHrvkB Ywk5q/aR+8+d4dGvTst7PzOxpNOwqaEEztZeA0fUmlCGzV7XweyvAb3J2YnRPqS3XIh/uu S51nui/CfPLXSAzIfhhGqRppHdsM7+pj/aJQunDArrCQrXIlV56RxLU35hIV2g== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1795102958; b=pkD6ET5Y6kBCfSaPG8t+CXBCY7ENb+7kqvLs6GhDDHmlkgNU+PLDckqIhW1pALiwV3rm6c +F01uNHLB+uxLNn7Kt5gMMnbDf4WdBQqDFxklYKPpU3ff0WYnFqG3Jma31hMxLaeIkA84C GGHjag2+jAdN/fyh7/t0WfCOO7gR7akn8D1o8IyDBjn/Yz2gGT0g3/l3kM+kA7QoFHeoMw dVktCRXYijwgmNSyz9iDYT0YSbHQIEvCPEy2fO2rPW7QpHdpaxgbVJYsJxqBvt5z6eGq/D R9bZB8EvFdCEHE77SOW7U4mFnG7BXMzWE3E3MiUQ8+49SvKEqo8JtI4Uwt0cRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1795102958; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=qArDrwOZbzNLcRoFyBC9HkL01wvILOuO1lIduePHq78=; b=rZ15amaVk+KGscsK1dkm0zOQg/p+zjrlhPqvr1jbxbFNv3gA06HDeQhCA3NUdNcYjw/4fi litiV1xUR2uei0RcSLpZauJ5cRqPeDdJv8p4SPEPl1wEFmny2SzU4jGQPKEcARkhuM5OzN yCUcrUKvl5K9tjrpNHaI8AejI3wv7Ji9hHVl7k3JUdF3i3K/TQZXVE9aI+0QvIR+uTUCYr fiAFaA0wrz8/c/PAu1cvuQkHgEwPMyLd3ZZt3cbUp8JKCfSVHxYdOXuK1bI1JQJmQgPeFC w5k0kyyWEICJjra15yUauyM/7l1npjiWdQp5g/bRO5FdV4RHM1CWGYnHpsxQ0g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4jVjQk2Vmpz16ZG for ; Thu, 19 Nov 2026 15:42:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id cd44 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Thu, 19 Nov 2026 15:42:38 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: John Baldwin Subject: git: 0c064ad61023 - stable/15 - cxgbe tls: Send final data as immediate when trailing waste is trimmed List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all X-BeenThere: dev-commits-src-all@freebsd.org List-Id: Commit messages for all branches of the src repository List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 0c064ad610236e1dfa8b064cd12d3d7c00766bf8 Auto-Submitted: auto-generated Date: Thu, 19 Nov 2026 15:42:38 +0000 Message-Id: <6aff18ee.cd44.5703de3d@gitrepo.freebsd.org> The branch stable/15 has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=0c064ad610236e1dfa8b064cd12d3d7c00766bf8 commit 0c064ad610236e1dfa8b064cd12d3d7c00766bf8 Author: John Baldwin AuthorDate: 2026-09-30 19:19:11 +0000 Commit: John Baldwin CommitDate: 2026-11-19 15:42:20 +0000 cxgbe tls: Send final data as immediate when trailing waste is trimmed When a request needs to drop data from the crypto output (via a split mode request), send the last 16 bytes of input as immediate data instead of via DSGL. Requests with small payloads (16 bytes or fewer) are now sent as immediate data only without any DSGL at all. Sponsored by: Chelsio Communications (cherry picked from commit 1bc81d3bb05505d1facc6884ee875ccb56e42977) --- sys/dev/cxgbe/adapter.h | 2 + sys/dev/cxgbe/crypto/t7_kern_tls.c | 197 ++++++++++++++++++++++++++++++++----- sys/dev/cxgbe/t4_main.c | 2 + sys/dev/cxgbe/t4_sge.c | 8 ++ 4 files changed, 185 insertions(+), 24 deletions(-) diff --git a/sys/dev/cxgbe/adapter.h b/sys/dev/cxgbe/adapter.h index b84c673f75ab..62d6c9212963 100644 --- a/sys/dev/cxgbe/adapter.h +++ b/sys/dev/cxgbe/adapter.h @@ -688,6 +688,8 @@ struct sge_txq { uint64_t kern_tls_partial_ghash; uint64_t kern_tls_splitmode; uint64_t kern_tls_trailer; + uint64_t kern_tls_imm_only; + uint64_t kern_tls_imm_last16; }; }; diff --git a/sys/dev/cxgbe/crypto/t7_kern_tls.c b/sys/dev/cxgbe/crypto/t7_kern_tls.c index 033ddbd586b8..ef0d0f577ffa 100644 --- a/sys/dev/cxgbe/crypto/t7_kern_tls.c +++ b/sys/dev/cxgbe/crypto/t7_kern_tls.c @@ -49,6 +49,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include "common/common.h" #include "common/t4_regs.h" @@ -634,6 +635,14 @@ ktls_wr_len(struct tlspcb *tlsp, struct mbuf *m, struct mbuf *m_tls) if (use_ghash) imm_len += ktls_gcm_aad_len(tlsp); + /* + * We might send 16 bytes of payload and a GHASH placeholder + * as immediate data. + */ + imm_len += 16; + if (use_ghash) + imm_len += AES_GMAC_HASH_LEN; + wr_len += roundup2(imm_len, 16); /* @@ -651,6 +660,14 @@ ktls_wr_len(struct tlspcb *tlsp, struct mbuf *m, struct mbuf *m_tls) MPASS(nsegs <= TX_SGL_SEGS); wr_len += ktls_sgl_size(nsegs); + /* + * Might send 16 bytes of immediate data after DSGL. This would + * be instead of 16 bytes of imm_len data above, so just add + * header. + */ + wr_len += sizeof(struct ulptx_idata); + wr_len = roundup2(wr_len, 16); + if (use_ghash) { /* AES-GCM records might return a partial hash. */ wr_len += sizeof(struct ulp_txpkt); @@ -1020,7 +1037,7 @@ write_split_mode_rx_phys(void *dst, struct mbuf *m, struct mbuf *m_tls, * add a 0 filled flit at the end. */ static void * -write_gl_to_buf(struct sglist *gl, caddr_t to) +write_gl_to_buf(struct sglist *gl, caddr_t to, bool more) { struct sglist_seg *seg; __be64 *flitp; @@ -1039,7 +1056,7 @@ write_gl_to_buf(struct sglist *gl, caddr_t to) usgl = (void *)flitp; usgl->cmd_nsge = htobe32(V_ULPTX_CMD(ULP_TX_SC_DSGL) | - V_ULPTX_NSGE(nsegs)); + V_ULP_TX_SC_MORE(more ? 1 : 0) | V_ULPTX_NSGE(nsegs)); usgl->len0 = htobe32(seg->ss_len); usgl->addr0 = htobe64(seg->ss_paddr); seg++; @@ -1051,6 +1068,8 @@ write_gl_to_buf(struct sglist *gl, caddr_t to) if (i & 1) usgl->sge[i / 2].len[1] = htobe32(0); flitp += nflits; + if (more) + return (flitp); if (nflits & 1) { MPASS(((uintptr_t)flitp) & 0xf); @@ -1178,6 +1197,57 @@ ktls_write_tunnel_packet(struct sge_txq *txq, void *dst, struct mbuf *m, return (ndesc); } +/* + * A variant of m_copydata()/m_unmapped_uiomove() that accepts an + * offset/length into the backing store of an EXTPG mbuf ignoring the + * leading offset in m_data and length in m_len. This is needed as + * KTLS requests sometimes need to access data from the TLS record + * that are not part of the requested data sent on the wire. + */ +static void +m_copyepgdata_raw(struct mbuf *m, int off, int len, caddr_t cp) +{ + int i, pglen, pgoff, seglen, segoff; + + M_ASSERTEXTPG(m); + if (m->m_epg_hdrlen != 0) { + if (off >= m->m_epg_hdrlen) { + off -= m->m_epg_hdrlen; + } else { + seglen = m->m_epg_hdrlen - off; + segoff = off; + seglen = min(seglen, len); + off = 0; + len -= seglen; + memcpy(cp, &m->m_epg_hdr[segoff], seglen); + cp += seglen; + } + } + pgoff = m->m_epg_1st_off; + for (i = 0; i < m->m_epg_npgs && len > 0; i++) { + pglen = m_epg_pagelen(m, i, pgoff); + if (off >= pglen) { + off -= pglen; + pgoff = 0; + continue; + } + seglen = pglen - off; + segoff = pgoff + off; + off = 0; + seglen = min(seglen, len); + len -= seglen; + memcpy(cp, (char *)PHYS_TO_DMAP(m->m_epg_pa[i]) + segoff, + seglen); + cp += seglen; + }; + if (len != 0) { + KASSERT((off + len) <= m->m_epg_trllen, + ("off + len > trail (%d + %d > %d)", off, len, + m->m_epg_trllen)); + memcpy(cp, &m->m_epg_trail[off], len); + } +} + static int ktls_write_tls_wr(struct tlspcb *tlsp, struct sge_txq *txq, void *dst, struct mbuf *m, struct tcphdr *tcp, struct mbuf *m_tls, @@ -1203,11 +1273,11 @@ ktls_write_tls_wr(struct tlspcb *tlsp, struct sge_txq *txq, u_int header_len, offset, plen, rlen, tlen; u_int imm_len, ndesc, nsegs, txpkt_lens[2], wr_len; u_int cpl_len, crypto_hdr_len, post_key_context_len; - u_int leading_waste, trailing_waste; + u_int leading_waste, trailing_waste, plen_dsgl_len; u_short ip_len; - bool inline_key, ghash_lcb, last_ghash_frag, last_wr, need_lso; - bool request_ghash, send_partial_ghash, short_record, split_mode; - bool using_scratch; + bool inline_key, ghash_lcb, ghash_placeholder_in_dsgl, last_ghash_frag; + bool last_wr, need_lso, request_ghash, send_partial_ghash, short_record; + bool split_mode, using_scratch; MPASS(tlsp->txq == txq); M_ASSERTEXTPG(m_tls); @@ -1404,8 +1474,32 @@ ktls_write_tls_wr(struct tlspcb *tlsp, struct sge_txq *txq, #endif } + /* + * If there is trailing waste, send the last 16 bytes of data + * as immediate data after the DSGL. + */ + ghash_placeholder_in_dsgl = last_ghash_frag; + plen_dsgl_len = plen; + if (trailing_waste != 0) { + if (plen <= 16) { + /* Just send entire payload as immediate data. */ + plen_dsgl_len = 0; + ghash_placeholder_in_dsgl = false; + } else if (last_ghash_frag) { + _Static_assert(AES_GMAC_HASH_LEN == 16, + "GMAC hash length mismatch"); + ghash_placeholder_in_dsgl = false; + } else { + plen_dsgl_len -= 16; + } +#ifdef VERBOSE_TRACES + CTR(KTR_CXGBE, "%s: %p plen_dsgl_len %u ghash in dsgl %u", + __func__, tlsp, plen_dsgl_len, ghash_placeholder_in_dsgl); +#endif + } + nsegs = sglist_count_mbuf_epg(m_tls, m_tls->m_epg_hdrlen + offset, - plen); + plen_dsgl_len); /* Determine if we need an LSO header. */ need_lso = (m_tls->m_len > mss); @@ -1443,8 +1537,20 @@ ktls_write_tls_wr(struct tlspcb *tlsp, struct sge_txq *txq, imm_len += ktls_gcm_aad_len(tlsp); } else if (tlsp->tls13) imm_len += sizeof(uint64_t); + if (plen_dsgl_len == 0) { + imm_len += plen; + if (last_ghash_frag) + imm_len += AES_GMAC_HASH_LEN; + } wr_len += roundup2(imm_len, 16); - wr_len += ktls_sgl_size(nsegs + (last_ghash_frag ? 1 : 0)); + if (plen_dsgl_len != 0) { + wr_len += ktls_sgl_size(nsegs + + (ghash_placeholder_in_dsgl ? 1 : 0)); + if (trailing_waste != 0) { + wr_len += sizeof(struct ulptx_idata); + wr_len += 16; + } + } wr_len = roundup2(wr_len, 16); txpkt_lens[0] = wr_len - sizeof(*wr); @@ -1497,7 +1603,7 @@ ktls_write_tls_wr(struct tlspcb *tlsp, struct sge_txq *txq, /* ULPTX_IDATA sub-command */ idata = (void *)(txpkt + 1); idata->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_IMM) | - V_ULP_TX_SC_MORE(1)); + V_ULP_TX_SC_MORE((!inline_key || plen_dsgl_len != 0) ? 1 : 0)); idata->len = sizeof(struct cpl_tx_sec_pdu); /* @@ -1721,10 +1827,13 @@ ktls_write_tls_wr(struct tlspcb *tlsp, struct sge_txq *txq, V_ULPTX_LEN16(tlsp->tx_key_info_size >> 4)); memrd->addr = htobe32(tlsp->tx_key_addr >> 5); - /* ULPTX_IDATA for CPL_TX_* and headers. */ + /* + * ULPTX_IDATA for CPL_TX_* and immediate data + * (including headers). + */ idata = (void *)(memrd + 1); idata->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_IMM) | - V_ULP_TX_SC_MORE(1)); + V_ULP_TX_SC_MORE(plen_dsgl_len != 0 ? 1 : 0)); idata->len = htobe32(post_key_context_len); out = (void *)(idata + 1); @@ -1838,6 +1947,17 @@ ktls_write_tls_wr(struct tlspcb *tlsp, struct sge_txq *txq, out += AES_BLOCK_LEN; } + if (plen_dsgl_len == 0) { + m_copyepgdata_raw(m_tls, m_tls->m_epg_hdrlen + offset, plen, + out); + out += plen; + if (last_ghash_frag) { + memset(out, 0, AES_GMAC_HASH_LEN); + out += AES_GMAC_HASH_LEN; + } + txq->kern_tls_imm_only++; + } + if (imm_len % 16 != 0) { if (imm_len % 8 != 0) { /* Zero pad to an 8-byte boundary. */ @@ -1852,31 +1972,60 @@ ktls_write_tls_wr(struct tlspcb *tlsp, struct sge_txq *txq, if (imm_len % 16 <= 8) { idata = (void *)out; idata->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_NOOP) | - V_ULP_TX_SC_MORE(1)); + V_ULP_TX_SC_MORE(plen_dsgl_len != 0 ? 1 : 0)); idata->len = htobe32(0); out = (void *)(idata + 1); } } /* SGL for record payload */ - sglist_reset(txq->gl); - if (sglist_append_mbuf_epg(txq->gl, m_tls, m_tls->m_epg_hdrlen + offset, - plen) != 0) { + if (plen_dsgl_len != 0) { + sglist_reset(txq->gl); + if (sglist_append_mbuf_epg(txq->gl, m_tls, + m_tls->m_epg_hdrlen + offset, plen_dsgl_len) != 0) { #ifdef INVARIANTS - panic("%s: failed to append sglist", __func__); + panic("%s: failed to append sglist", __func__); #endif - } - KASSERT(txq->gl->sg_nseg == nsegs, ("%s: sg_nseg %u != nsegs %u", - __func__, txq->gl->sg_nseg, nsegs)); - if (last_ghash_frag) { - if (sglist_append_phys(txq->gl, zero_buffer_pa, - AES_GMAC_HASH_LEN) != 0) { + } + KASSERT(txq->gl->sg_nseg == nsegs, + ("%s: sg_nseg %u != nsegs %u", __func__, txq->gl->sg_nseg, + nsegs)); + if (ghash_placeholder_in_dsgl) { + if (sglist_append_phys(txq->gl, zero_buffer_pa, + AES_GMAC_HASH_LEN) != 0) { #ifdef INVARIANTS - panic("%s: failed to append sglist (2)", __func__); + panic("%s: failed to append sglist (2)", + __func__); #endif + } + } + out = write_gl_to_buf(txq->gl, out, trailing_waste != 0); + if (trailing_waste != 0) { + idata = (void *)out; + idata->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_IMM)); + idata->len = htobe32(16); + out = (void *)(idata + 1); + + if (last_ghash_frag) { + MPASS(!ghash_placeholder_in_dsgl); + MPASS(plen == plen_dsgl_len); + memset(out, 0, AES_GMAC_HASH_LEN); + out += AES_GMAC_HASH_LEN; + } else { + MPASS(plen_dsgl_len == plen - 16); + m_copyepgdata_raw(m_tls, m_tls->m_epg_hdrlen + + offset + plen_dsgl_len, 16, out); + out += 16; + } + txq->kern_tls_imm_last16++; + + if (!__is_aligned(out, 16)) { + MPASS((uintptr_t)out % 16 == 8); + memset(out, 0, 8); + out += 8; + } } } - out = write_gl_to_buf(txq->gl, out); KASSERT((char *)out - (char *)(wr + 1) == roundup2(txpkt_lens[0], 16), ("%s: txpkts_len[0] mismatch: %td vs %u", __func__, diff --git a/sys/dev/cxgbe/t4_main.c b/sys/dev/cxgbe/t4_main.c index 77ff10bc861b..3cdfbc15ceec 100644 --- a/sys/dev/cxgbe/t4_main.c +++ b/sys/dev/cxgbe/t4_main.c @@ -13026,6 +13026,8 @@ clear_stats(struct adapter *sc, u_int port_id) txq->kern_tls_partial_ghash = 0; txq->kern_tls_splitmode = 0; txq->kern_tls_trailer = 0; + txq->kern_tls_imm_only = 0; + txq->kern_tls_imm_last16 = 0; } mp_ring_reset_stats(txq->r); } diff --git a/sys/dev/cxgbe/t4_sge.c b/sys/dev/cxgbe/t4_sge.c index 2f9cb1a4ebb5..0495ea442a97 100644 --- a/sys/dev/cxgbe/t4_sge.c +++ b/sys/dev/cxgbe/t4_sge.c @@ -4898,6 +4898,14 @@ add_txq_sysctls(struct vi_info *vi, struct sysctl_ctx_list *ctx, "kern_tls_trailer", CTLFLAG_RD, &txq->kern_tls_trailer, "# of NIC TLS trailer-only packets transmitted"); + SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, + "kern_tls_imm_only", CTLFLAG_RD, + &txq->kern_tls_imm_only, + "# of NIC TLS records with only immediate payload"); + SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, + "kern_tls_imm_last16", CTLFLAG_RD, + &txq->kern_tls_imm_last16, + "# of NIC TLS records with last 16 bytes as immediate data"); } SYSCTL_ADD_UQUAD(ctx, children, OID_AUTO, "kern_tls_cbc", CTLFLAG_RD, &txq->kern_tls_cbc,