svn commit: r354974 - in head/sys: conf dev/cxgbe dev/cxgbe/common dev/cxgbe/crypto dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/if_cxgbe
John Baldwin
jhb at FreeBSD.org
Thu Nov 21 19:30:33 UTC 2019
Author: jhb
Date: Thu Nov 21 19:30:31 2019
New Revision: 354974
URL: https://svnweb.freebsd.org/changeset/base/354974
Log:
NIC KTLS for Chelsio T6 adapters.
This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.
NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:
1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.
2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.
3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.
Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.
TCP packets for TLS requests are broken down into the following
classes (with associated counters):
- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).
- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.
- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.
In addition, other per-txq sysctls are provided:
- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.
- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.
- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.
- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.
- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.
- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.
To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:
hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962
Added:
head/sys/dev/cxgbe/crypto/t4_kern_tls.c (contents, props changed)
head/sys/dev/cxgbe/firmware/t6fw_cfg_kern_tls.txt (contents, props changed)
Modified:
head/sys/conf/files
head/sys/dev/cxgbe/adapter.h
head/sys/dev/cxgbe/common/t4_msg.h
head/sys/dev/cxgbe/common/t4_regs.h
head/sys/dev/cxgbe/common/t4_tcb.h
head/sys/dev/cxgbe/offload.h
head/sys/dev/cxgbe/t4_l2t.c
head/sys/dev/cxgbe/t4_l2t.h
head/sys/dev/cxgbe/t4_main.c
head/sys/dev/cxgbe/t4_sge.c
head/sys/dev/cxgbe/tom/t4_connect.c
head/sys/dev/cxgbe/tom/t4_listen.c
head/sys/modules/cxgbe/if_cxgbe/Makefile
Modified: head/sys/conf/files
==============================================================================
--- head/sys/conf/files Thu Nov 21 19:13:05 2019 (r354973)
+++ head/sys/conf/files Thu Nov 21 19:30:31 2019 (r354974)
@@ -1423,6 +1423,8 @@ dev/cxgbe/common/t4_hw.c optional cxgbe pci \
compile-with "${NORMAL_C} -I$S/dev/cxgbe"
dev/cxgbe/common/t4vf_hw.c optional cxgbev pci \
compile-with "${NORMAL_C} -I$S/dev/cxgbe"
+dev/cxgbe/crypto/t4_kern_tls.c optional cxgbe pci kern_tls \
+ compile-with "${NORMAL_C} -I$S/dev/cxgbe"
dev/cxgbe/crypto/t4_keyctx.c optional cxgbe pci \
compile-with "${NORMAL_C} -I$S/dev/cxgbe"
dev/cxgbe/cudbg/cudbg_common.c optional cxgbe \
Modified: head/sys/dev/cxgbe/adapter.h
==============================================================================
--- head/sys/dev/cxgbe/adapter.h Thu Nov 21 19:13:05 2019 (r354973)
+++ head/sys/dev/cxgbe/adapter.h Thu Nov 21 19:30:31 2019 (r354974)
@@ -35,6 +35,7 @@
#include <sys/kernel.h>
#include <sys/bus.h>
+#include <sys/counter.h>
#include <sys/rman.h>
#include <sys/types.h>
#include <sys/lock.h>
@@ -158,6 +159,7 @@ enum {
ADAP_ERR = (1 << 5),
BUF_PACKING_OK = (1 << 6),
IS_VF = (1 << 7),
+ KERN_TLS_OK = (1 << 8),
CXGBE_BUSY = (1 << 9),
@@ -380,7 +382,7 @@ enum {
CPL_COOKIE_TOM,
CPL_COOKIE_HASHFILTER,
CPL_COOKIE_ETHOFLD,
- CPL_COOKIE_AVAILABLE3,
+ CPL_COOKIE_KERN_TLS,
NUM_CPL_COOKIES = 8 /* Limited by M_COOKIE. Do not increase. */
};
@@ -582,8 +584,25 @@ struct sge_txq {
uint64_t txpkts0_pkts; /* # of frames in type0 coalesced tx WRs */
uint64_t txpkts1_pkts; /* # of frames in type1 coalesced tx WRs */
uint64_t raw_wrs; /* # of raw work requests (alloc_wr_mbuf) */
+ uint64_t tls_wrs; /* # of TLS work requests */
+ uint64_t kern_tls_records;
+ uint64_t kern_tls_short;
+ uint64_t kern_tls_partial;
+ uint64_t kern_tls_full;
+ uint64_t kern_tls_octets;
+ uint64_t kern_tls_waste;
+ uint64_t kern_tls_options;
+ uint64_t kern_tls_header;
+ uint64_t kern_tls_fin;
+ uint64_t kern_tls_fin_short;
+ uint64_t kern_tls_cbc;
+ uint64_t kern_tls_gcm;
+
/* stats for not-that-common events */
+
+ /* Optional scratch space for constructing work requests. */
+ uint8_t ss[SGE_MAX_WR_LEN] __aligned(16);
} __aligned(CACHE_LINE_SIZE);
/* rxq: SGE ingress queue + SGE free list + miscellaneous items */
@@ -840,6 +859,7 @@ struct adapter {
struct smt_data *smt; /* Source MAC Table */
struct tid_info tids;
vmem_t *key_map;
+ struct tls_tunables tlst;
uint8_t doorbells;
int offload_map; /* ports with IFCAP_TOE enabled */
@@ -897,6 +917,8 @@ struct adapter {
int last_op_flags;
int swintr;
+
+ struct callout ktls_tick;
};
#define ADAPTER_LOCK(sc) mtx_lock(&(sc)->sc_lock)
@@ -1168,6 +1190,18 @@ int cxgbe_media_change(struct ifnet *);
void cxgbe_media_status(struct ifnet *, struct ifmediareq *);
bool t4_os_dump_cimla(struct adapter *, int, bool);
void t4_os_dump_devlog(struct adapter *);
+
+#ifdef KERN_TLS
+/* t4_kern_tls.c */
+int cxgbe_tls_tag_alloc(struct ifnet *, union if_snd_tag_alloc_params *,
+ struct m_snd_tag **);
+void cxgbe_tls_tag_free(struct m_snd_tag *);
+void t6_ktls_modload(void);
+void t6_ktls_modunload(void);
+int t6_ktls_try(struct ifnet *, struct socket *, struct ktls_session *);
+int t6_ktls_parse_pkt(struct mbuf *, int *, int *);
+int t6_ktls_write_wr(struct sge_txq *, void *, struct mbuf *, u_int, u_int);
+#endif
/* t4_keyctx.c */
struct auth_hash;
Modified: head/sys/dev/cxgbe/common/t4_msg.h
==============================================================================
--- head/sys/dev/cxgbe/common/t4_msg.h Thu Nov 21 19:13:05 2019 (r354973)
+++ head/sys/dev/cxgbe/common/t4_msg.h Thu Nov 21 19:30:31 2019 (r354974)
@@ -1158,6 +1158,17 @@ struct cpl_tx_data {
__be32 flags;
};
+/* cpl_tx_data.len fields */
+#define S_TX_DATA_MSS 16
+#define M_TX_DATA_MSS 0xFFFF
+#define V_TX_DATA_MSS(x) ((x) << S_TX_DATA_MSS)
+#define G_TX_DATA_MSS(x) (((x) >> S_TX_DATA_MSS) & M_TX_DATA_MSS)
+
+#define S_TX_LENGTH 0
+#define M_TX_LENGTH 0xFFFF
+#define V_TX_LENGTH(x) ((x) << S_TX_LENGTH)
+#define G_TX_LENGTH(x) (((x) >> S_TX_LENGTH) & M_TX_LENGTH)
+
/* cpl_tx_data.flags fields */
#define S_TX_PROXY 5
#define V_TX_PROXY(x) ((x) << S_TX_PROXY)
@@ -1204,6 +1215,14 @@ struct cpl_tx_data {
#define S_T6_TX_FORCE 20
#define V_T6_TX_FORCE(x) ((x) << S_T6_TX_FORCE)
#define F_T6_TX_FORCE V_T6_TX_FORCE(1U)
+
+#define S_TX_BYPASS 21
+#define V_TX_BYPASS(x) ((x) << S_TX_BYPASS)
+#define F_TX_BYPASS V_TX_BYPASS(1U)
+
+#define S_TX_PUSH 22
+#define V_TX_PUSH(x) ((x) << S_TX_PUSH)
+#define F_TX_PUSH V_TX_PUSH(1U)
/* additional tx_data_wr.flags fields */
#define S_TX_CPU_IDX 0
Modified: head/sys/dev/cxgbe/common/t4_regs.h
==============================================================================
--- head/sys/dev/cxgbe/common/t4_regs.h Thu Nov 21 19:13:05 2019 (r354973)
+++ head/sys/dev/cxgbe/common/t4_regs.h Thu Nov 21 19:30:31 2019 (r354974)
@@ -22617,6 +22617,10 @@
#define V_TXPDUSIZEADJ(x) ((x) << S_TXPDUSIZEADJ)
#define G_TXPDUSIZEADJ(x) (((x) >> S_TXPDUSIZEADJ) & M_TXPDUSIZEADJ)
+#define S_ENABLECBYP 21
+#define V_ENABLECBYP(x) ((x) << S_ENABLECBYP)
+#define F_ENABLECBYP V_ENABLECBYP(1U)
+
#define S_LIMITEDTRANSMIT 20
#define M_LIMITEDTRANSMIT 0xfU
#define V_LIMITEDTRANSMIT(x) ((x) << S_LIMITEDTRANSMIT)
Modified: head/sys/dev/cxgbe/common/t4_tcb.h
==============================================================================
--- head/sys/dev/cxgbe/common/t4_tcb.h Thu Nov 21 19:13:05 2019 (r354973)
+++ head/sys/dev/cxgbe/common/t4_tcb.h Thu Nov 21 19:30:31 2019 (r354974)
@@ -753,6 +753,9 @@
#define S_TF_CCTRL_RFR 62
#define V_TF_CCTRL_RFR(x) ((__u64)(x) << S_TF_CCTRL_RFR)
+#define S_TF_CORE_BYPASS 63
+#define V_TF_CORE_BYPASS(x) ((__u64)(x) << S_TF_CORE_BYPASS)
+
#define S_TF_DDP_INDICATE_OUT 16
#define V_TF_DDP_INDICATE_OUT(x) ((x) << S_TF_DDP_INDICATE_OUT)
Added: head/sys/dev/cxgbe/crypto/t4_kern_tls.c
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ head/sys/dev/cxgbe/crypto/t4_kern_tls.c Thu Nov 21 19:30:31 2019 (r354974)
@@ -0,0 +1,2400 @@
+/*-
+ * SPDX-License-Identifier: BSD-2-Clause-FreeBSD
+ *
+ * Copyright (c) 2018-2019 Chelsio Communications, Inc.
+ * All rights reserved.
+ * Written by: John Baldwin <jhb at FreeBSD.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "opt_inet.h"
+#include "opt_inet6.h"
+#include "opt_kern_tls.h"
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <sys/param.h>
+#include <sys/ktr.h>
+#include <sys/ktls.h>
+#include <sys/sglist.h>
+#include <sys/socket.h>
+#include <sys/socketvar.h>
+#include <sys/sockbuf.h>
+#include <netinet/in.h>
+#include <netinet/in_pcb.h>
+#include <netinet/ip.h>
+#include <netinet/ip6.h>
+#include <netinet/tcp_var.h>
+#include <opencrypto/cryptodev.h>
+#include <opencrypto/xform.h>
+
+#include "common/common.h"
+#include "common/t4_regs.h"
+#include "common/t4_regs_values.h"
+#include "common/t4_tcb.h"
+#include "t4_l2t.h"
+#include "t4_clip.h"
+#include "t4_mp_ring.h"
+#include "crypto/t4_crypto.h"
+
+#if defined(INET) || defined(INET6)
+
+#define SALT_SIZE 4
+
+#define GCM_TAG_SIZE 16
+#define TLS_HEADER_LENGTH 5
+
+#define TLS_KEY_CONTEXT_SZ roundup2(sizeof(struct tls_keyctx), 32)
+
+struct tls_scmd {
+ __be32 seqno_numivs;
+ __be32 ivgen_hdrlen;
+};
+
+struct tls_key_req {
+ /* FW_ULPTX_WR */
+ __be32 wr_hi;
+ __be32 wr_mid;
+ __be32 ftid;
+ __u8 reneg_to_write_rx;
+ __u8 protocol;
+ __be16 mfs;
+ /* master command */
+ __be32 cmd;
+ __be32 len16; /* command length */
+ __be32 dlen; /* data length in 32-byte units */
+ __be32 kaddr;
+ /* sub-command */
+ __be32 sc_more;
+ __be32 sc_len;
+}__packed;
+
+struct tls_keyctx {
+ struct tx_keyctx_hdr {
+ __u8 ctxlen;
+ __u8 r2;
+ __be16 dualck_to_txvalid;
+ __u8 txsalt[4];
+ __be64 r5;
+ } txhdr;
+ struct keys {
+ __u8 edkey[32];
+ __u8 ipad[64];
+ __u8 opad[64];
+ } keys;
+};
+
+#define S_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT 11
+#define M_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT 0x1
+#define V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(x) \
+ ((x) << S_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT)
+#define G_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(x) \
+ (((x) >> S_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT) & \
+ M_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT)
+#define F_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT \
+ V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1U)
+
+#define S_TLS_KEYCTX_TX_WR_SALT_PRESENT 10
+#define M_TLS_KEYCTX_TX_WR_SALT_PRESENT 0x1
+#define V_TLS_KEYCTX_TX_WR_SALT_PRESENT(x) \
+ ((x) << S_TLS_KEYCTX_TX_WR_SALT_PRESENT)
+#define G_TLS_KEYCTX_TX_WR_SALT_PRESENT(x) \
+ (((x) >> S_TLS_KEYCTX_TX_WR_SALT_PRESENT) & \
+ M_TLS_KEYCTX_TX_WR_SALT_PRESENT)
+#define F_TLS_KEYCTX_TX_WR_SALT_PRESENT \
+ V_TLS_KEYCTX_TX_WR_SALT_PRESENT(1U)
+
+#define S_TLS_KEYCTX_TX_WR_TXCK_SIZE 6
+#define M_TLS_KEYCTX_TX_WR_TXCK_SIZE 0xf
+#define V_TLS_KEYCTX_TX_WR_TXCK_SIZE(x) \
+ ((x) << S_TLS_KEYCTX_TX_WR_TXCK_SIZE)
+#define G_TLS_KEYCTX_TX_WR_TXCK_SIZE(x) \
+ (((x) >> S_TLS_KEYCTX_TX_WR_TXCK_SIZE) & \
+ M_TLS_KEYCTX_TX_WR_TXCK_SIZE)
+
+#define S_TLS_KEYCTX_TX_WR_TXMK_SIZE 2
+#define M_TLS_KEYCTX_TX_WR_TXMK_SIZE 0xf
+#define V_TLS_KEYCTX_TX_WR_TXMK_SIZE(x) \
+ ((x) << S_TLS_KEYCTX_TX_WR_TXMK_SIZE)
+#define G_TLS_KEYCTX_TX_WR_TXMK_SIZE(x) \
+ (((x) >> S_TLS_KEYCTX_TX_WR_TXMK_SIZE) & \
+ M_TLS_KEYCTX_TX_WR_TXMK_SIZE)
+
+#define S_TLS_KEYCTX_TX_WR_TXVALID 0
+#define M_TLS_KEYCTX_TX_WR_TXVALID 0x1
+#define V_TLS_KEYCTX_TX_WR_TXVALID(x) \
+ ((x) << S_TLS_KEYCTX_TX_WR_TXVALID)
+#define G_TLS_KEYCTX_TX_WR_TXVALID(x) \
+ (((x) >> S_TLS_KEYCTX_TX_WR_TXVALID) & M_TLS_KEYCTX_TX_WR_TXVALID)
+#define F_TLS_KEYCTX_TX_WR_TXVALID V_TLS_KEYCTX_TX_WR_TXVALID(1U)
+
+/* Key Context Programming Operation type */
+#define KEY_WRITE_RX 0x1
+#define KEY_WRITE_TX 0x2
+#define KEY_DELETE_RX 0x4
+#define KEY_DELETE_TX 0x8
+
+struct tlspcb {
+ struct cxgbe_snd_tag com;
+ struct vi_info *vi; /* virtual interface */
+ struct adapter *sc;
+ struct l2t_entry *l2te; /* L2 table entry used by this connection */
+ int tid; /* Connection identifier */
+
+ int tx_key_addr;
+ bool inline_key;
+ bool using_timestamps;
+ unsigned char enc_mode;
+
+ struct tls_scmd scmd0;
+ struct tls_scmd scmd0_short;
+
+ unsigned int tx_key_info_size;
+
+ uint32_t prev_seq;
+ uint32_t prev_ack;
+ uint32_t prev_tsecr;
+ uint16_t prev_win;
+ uint16_t prev_mss;
+
+ /* Only used outside of setup and teardown when using inline keys. */
+ struct tls_keyctx keyctx;
+
+ /* Fields only used during setup and teardown. */
+ struct inpcb *inp; /* backpointer to host stack's PCB */
+ struct sge_txq *txq;
+ struct sge_wrq *ctrlq;
+ struct clip_entry *ce; /* CLIP table entry used by this tid */
+
+ unsigned char auth_mode;
+ unsigned char hmac_ctrl;
+ unsigned char mac_first;
+ unsigned char iv_size;
+
+ unsigned int frag_size;
+ unsigned int cipher_secret_size;
+ int proto_ver;
+
+ bool open_pending;
+};
+
+static int ktls_setup_keys(struct tlspcb *tlsp,
+ const struct ktls_session *tls, struct sge_txq *txq);
+
+static inline struct tlspcb *
+mst_to_tls(struct m_snd_tag *t)
+{
+ return ((struct tlspcb *)mst_to_cst(t));
+}
+
+/* XXX: There are similar versions of these two in tom/t4_tls.c. */
+static int
+get_new_keyid(struct tlspcb *tlsp)
+{
+ vmem_addr_t addr;
+
+ if (vmem_alloc(tlsp->sc->key_map, TLS_KEY_CONTEXT_SZ,
+ M_NOWAIT | M_FIRSTFIT, &addr) != 0)
+ return (-1);
+
+ return (addr);
+}
+
+static void
+free_keyid(struct tlspcb *tlsp, int keyid)
+{
+
+ CTR3(KTR_CXGBE, "%s: tid %d key addr %#x", __func__, tlsp->tid, keyid);
+ vmem_free(tlsp->sc->key_map, keyid, TLS_KEY_CONTEXT_SZ);
+}
+
+static struct tlspcb *
+alloc_tlspcb(struct ifnet *ifp, struct vi_info *vi, int flags)
+{
+ struct port_info *pi = vi->pi;
+ struct adapter *sc = pi->adapter;
+ struct tlspcb *tlsp;
+
+ tlsp = malloc(sizeof(*tlsp), M_CXGBE, M_ZERO | flags);
+ if (tlsp == NULL)
+ return (NULL);
+
+ cxgbe_snd_tag_init(&tlsp->com, ifp, IF_SND_TAG_TYPE_TLS);
+ tlsp->vi = vi;
+ tlsp->sc = sc;
+ tlsp->ctrlq = &sc->sge.ctrlq[pi->port_id];
+ tlsp->tid = -1;
+ tlsp->tx_key_addr = -1;
+
+ return (tlsp);
+}
+
+static void
+init_ktls_key_params(struct tlspcb *tlsp, const struct ktls_session *tls)
+{
+ int mac_key_size;
+
+ if (tls->params.tls_vminor == TLS_MINOR_VER_ONE)
+ tlsp->proto_ver = SCMD_PROTO_VERSION_TLS_1_1;
+ else
+ tlsp->proto_ver = SCMD_PROTO_VERSION_TLS_1_2;
+ tlsp->cipher_secret_size = tls->params.cipher_key_len;
+ tlsp->tx_key_info_size = sizeof(struct tx_keyctx_hdr) +
+ tlsp->cipher_secret_size;
+ if (tls->params.cipher_algorithm == CRYPTO_AES_NIST_GCM_16) {
+ tlsp->auth_mode = SCMD_AUTH_MODE_GHASH;
+ tlsp->enc_mode = SCMD_CIPH_MODE_AES_GCM;
+ tlsp->iv_size = 4;
+ tlsp->mac_first = 0;
+ tlsp->hmac_ctrl = SCMD_HMAC_CTRL_NOP;
+ tlsp->tx_key_info_size += GMAC_BLOCK_LEN;
+ } else {
+ switch (tls->params.auth_algorithm) {
+ case CRYPTO_SHA1_HMAC:
+ mac_key_size = roundup2(SHA1_HASH_LEN, 16);
+ tlsp->auth_mode = SCMD_AUTH_MODE_SHA1;
+ break;
+ case CRYPTO_SHA2_256_HMAC:
+ mac_key_size = SHA2_256_HASH_LEN;
+ tlsp->auth_mode = SCMD_AUTH_MODE_SHA256;
+ break;
+ case CRYPTO_SHA2_384_HMAC:
+ mac_key_size = SHA2_512_HASH_LEN;
+ tlsp->auth_mode = SCMD_AUTH_MODE_SHA512_384;
+ break;
+ }
+ tlsp->enc_mode = SCMD_CIPH_MODE_AES_CBC;
+ tlsp->iv_size = 8; /* for CBC, iv is 16B, unit of 2B */
+ tlsp->mac_first = 1;
+ tlsp->hmac_ctrl = SCMD_HMAC_CTRL_NO_TRUNC;
+ tlsp->tx_key_info_size += mac_key_size * 2;
+ }
+
+ tlsp->frag_size = tls->params.max_frame_len;
+}
+
+static int
+ktls_act_open_cpl_size(bool isipv6)
+{
+
+ if (isipv6)
+ return (sizeof(struct cpl_t6_act_open_req6));
+ else
+ return (sizeof(struct cpl_t6_act_open_req));
+}
+
+static void
+mk_ktls_act_open_req(struct adapter *sc, struct vi_info *vi, struct inpcb *inp,
+ struct tlspcb *tlsp, int atid, void *dst)
+{
+ struct tcpcb *tp = intotcpcb(inp);
+ struct cpl_t6_act_open_req *cpl6;
+ struct cpl_act_open_req *cpl;
+ uint64_t options;
+ int qid_atid;
+
+ cpl6 = dst;
+ cpl = (struct cpl_act_open_req *)cpl6;
+ INIT_TP_WR(cpl6, 0);
+ qid_atid = V_TID_QID(sc->sge.fwq.abs_id) | V_TID_TID(atid) |
+ V_TID_COOKIE(CPL_COOKIE_KERN_TLS);
+ OPCODE_TID(cpl) = htobe32(MK_OPCODE_TID(CPL_ACT_OPEN_REQ,
+ qid_atid));
+ inp_4tuple_get(inp, &cpl->local_ip, &cpl->local_port,
+ &cpl->peer_ip, &cpl->peer_port);
+
+ options = F_TCAM_BYPASS | V_ULP_MODE(ULP_MODE_NONE);
+ options |= V_SMAC_SEL(vi->smt_idx) | V_TX_CHAN(vi->pi->tx_chan);
+ options |= F_NON_OFFLOAD;
+ cpl->opt0 = htobe64(options);
+
+ options = V_TX_QUEUE(sc->params.tp.tx_modq[vi->pi->tx_chan]);
+ if (tp->t_flags & TF_REQ_TSTMP)
+ options |= F_TSTAMPS_EN;
+ cpl->opt2 = htobe32(options);
+}
+
+static void
+mk_ktls_act_open_req6(struct adapter *sc, struct vi_info *vi,
+ struct inpcb *inp, struct tlspcb *tlsp, int atid, void *dst)
+{
+ struct tcpcb *tp = intotcpcb(inp);
+ struct cpl_t6_act_open_req6 *cpl6;
+ struct cpl_act_open_req6 *cpl;
+ uint64_t options;
+ int qid_atid;
+
+ cpl6 = dst;
+ cpl = (struct cpl_act_open_req6 *)cpl6;
+ INIT_TP_WR(cpl6, 0);
+ qid_atid = V_TID_QID(sc->sge.fwq.abs_id) | V_TID_TID(atid) |
+ V_TID_COOKIE(CPL_COOKIE_KERN_TLS);
+ OPCODE_TID(cpl) = htobe32(MK_OPCODE_TID(CPL_ACT_OPEN_REQ6,
+ qid_atid));
+ cpl->local_port = inp->inp_lport;
+ cpl->local_ip_hi = *(uint64_t *)&inp->in6p_laddr.s6_addr[0];
+ cpl->local_ip_lo = *(uint64_t *)&inp->in6p_laddr.s6_addr[8];
+ cpl->peer_port = inp->inp_fport;
+ cpl->peer_ip_hi = *(uint64_t *)&inp->in6p_faddr.s6_addr[0];
+ cpl->peer_ip_lo = *(uint64_t *)&inp->in6p_faddr.s6_addr[8];
+
+ options = F_TCAM_BYPASS | V_ULP_MODE(ULP_MODE_NONE);
+ options |= V_SMAC_SEL(vi->smt_idx) | V_TX_CHAN(vi->pi->tx_chan);
+ options |= F_NON_OFFLOAD;
+ cpl->opt0 = htobe64(options);
+
+ options = V_TX_QUEUE(sc->params.tp.tx_modq[vi->pi->tx_chan]);
+ if (tp->t_flags & TF_REQ_TSTMP)
+ options |= F_TSTAMPS_EN;
+ cpl->opt2 = htobe32(options);
+}
+
+static int
+send_ktls_act_open_req(struct adapter *sc, struct vi_info *vi,
+ struct inpcb *inp, struct tlspcb *tlsp, int atid)
+{
+ struct wrqe *wr;
+ bool isipv6;
+
+ isipv6 = (inp->inp_vflag & INP_IPV6) != 0;
+ if (isipv6) {
+ tlsp->ce = t4_hold_lip(sc, &inp->in6p_laddr, NULL);
+ if (tlsp->ce == NULL)
+ return (ENOENT);
+ }
+
+ wr = alloc_wrqe(ktls_act_open_cpl_size(isipv6), tlsp->ctrlq);
+ if (wr == NULL) {
+ CTR2(KTR_CXGBE, "%s: atid %d failed to alloc WR", __func__,
+ atid);
+ return (ENOMEM);
+ }
+
+ if (isipv6)
+ mk_ktls_act_open_req6(sc, vi, inp, tlsp, atid, wrtod(wr));
+ else
+ mk_ktls_act_open_req(sc, vi, inp, tlsp, atid, wrtod(wr));
+
+ tlsp->open_pending = true;
+ t4_wrq_tx(sc, wr);
+ return (0);
+}
+
+static int
+ktls_act_open_rpl(struct sge_iq *iq, const struct rss_header *rss,
+ struct mbuf *m)
+{
+ struct adapter *sc = iq->adapter;
+ const struct cpl_act_open_rpl *cpl = (const void *)(rss + 1);
+ u_int atid = G_TID_TID(G_AOPEN_ATID(be32toh(cpl->atid_status)));
+ u_int status = G_AOPEN_STATUS(be32toh(cpl->atid_status));
+ struct tlspcb *tlsp = lookup_atid(sc, atid);
+ struct inpcb *inp = tlsp->inp;
+
+ CTR3(KTR_CXGBE, "%s: atid %d status %d", __func__, atid, status);
+ free_atid(sc, atid);
+ if (status == 0)
+ tlsp->tid = GET_TID(cpl);
+
+ INP_WLOCK(inp);
+ tlsp->open_pending = false;
+ wakeup(tlsp);
+ INP_WUNLOCK(inp);
+ return (0);
+}
+
+/* SET_TCB_FIELD sent as a ULP command looks like this */
+#define LEN__SET_TCB_FIELD_ULP (sizeof(struct ulp_txpkt) + \
+ sizeof(struct ulptx_idata) + sizeof(struct cpl_set_tcb_field_core))
+
+_Static_assert((LEN__SET_TCB_FIELD_ULP + sizeof(struct ulptx_idata)) % 16 == 0,
+ "CPL_SET_TCB_FIELD ULP command not 16-byte aligned");
+
+static void
+write_set_tcb_field_ulp(struct tlspcb *tlsp, void *dst, struct sge_txq *txq,
+ uint16_t word, uint64_t mask, uint64_t val)
+{
+ struct ulp_txpkt *txpkt;
+ struct ulptx_idata *idata;
+ struct cpl_set_tcb_field_core *cpl;
+
+ /* ULP_TXPKT */
+ txpkt = dst;
+ txpkt->cmd_dest = htobe32(V_ULPTX_CMD(ULP_TX_PKT) |
+ V_ULP_TXPKT_DATAMODIFY(0) |
+ V_ULP_TXPKT_CHANNELID(tlsp->vi->pi->port_id) | V_ULP_TXPKT_DEST(0) |
+ V_ULP_TXPKT_FID(txq->eq.cntxt_id) | V_ULP_TXPKT_RO(1));
+ txpkt->len = htobe32(howmany(LEN__SET_TCB_FIELD_ULP, 16));
+
+ /* ULPTX_IDATA sub-command */
+ idata = (struct ulptx_idata *)(txpkt + 1);
+ idata->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_IMM));
+ idata->len = htobe32(sizeof(*cpl));
+
+ /* CPL_SET_TCB_FIELD */
+ cpl = (struct cpl_set_tcb_field_core *)(idata + 1);
+ OPCODE_TID(cpl) = htobe32(MK_OPCODE_TID(CPL_SET_TCB_FIELD, tlsp->tid));
+ cpl->reply_ctrl = htobe16(F_NO_REPLY);
+ cpl->word_cookie = htobe16(V_WORD(word));
+ cpl->mask = htobe64(mask);
+ cpl->val = htobe64(val);
+
+ /* ULPTX_NOOP */
+ idata = (struct ulptx_idata *)(cpl + 1);
+ idata->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_NOOP));
+ idata->len = htobe32(0);
+}
+
+static int
+ktls_set_tcb_fields(struct tlspcb *tlsp, struct tcpcb *tp, struct sge_txq *txq)
+{
+ struct fw_ulptx_wr *wr;
+ struct mbuf *m;
+ char *dst;
+ void *items[1];
+ int error, len;
+
+ len = sizeof(*wr) + 3 * roundup2(LEN__SET_TCB_FIELD_ULP, 16);
+ if (tp->t_flags & TF_REQ_TSTMP)
+ len += roundup2(LEN__SET_TCB_FIELD_ULP, 16);
+ m = alloc_wr_mbuf(len, M_NOWAIT);
+ if (m == NULL) {
+ CTR2(KTR_CXGBE, "%s: tid %d failed to alloc WR mbuf", __func__,
+ tlsp->tid);
+ return (ENOMEM);
+ }
+ m->m_pkthdr.snd_tag = m_snd_tag_ref(&tlsp->com.com);
+ m->m_pkthdr.csum_flags |= CSUM_SND_TAG;
+
+ /* FW_ULPTX_WR */
+ wr = mtod(m, void *);
+ wr->op_to_compl = htobe32(V_FW_WR_OP(FW_ULPTX_WR));
+ wr->flowid_len16 = htobe32(F_FW_ULPTX_WR_DATA |
+ V_FW_WR_LEN16(len / 16));
+ wr->cookie = 0;
+ dst = (char *)(wr + 1);
+
+ /* Clear TF_NON_OFFLOAD and set TF_CORE_BYPASS */
+ write_set_tcb_field_ulp(tlsp, dst, txq, W_TCB_T_FLAGS,
+ V_TCB_T_FLAGS(V_TF_CORE_BYPASS(1) | V_TF_NON_OFFLOAD(1)),
+ V_TCB_T_FLAGS(V_TF_CORE_BYPASS(1)));
+ dst += roundup2(LEN__SET_TCB_FIELD_ULP, 16);
+
+ /* Clear the SND_UNA_RAW, SND_NXT_RAW, and SND_MAX_RAW offsets. */
+ write_set_tcb_field_ulp(tlsp, dst, txq, W_TCB_SND_UNA_RAW,
+ V_TCB_SND_NXT_RAW(M_TCB_SND_NXT_RAW) |
+ V_TCB_SND_UNA_RAW(M_TCB_SND_UNA_RAW),
+ V_TCB_SND_NXT_RAW(0) | V_TCB_SND_UNA_RAW(0));
+ dst += roundup2(LEN__SET_TCB_FIELD_ULP, 16);
+
+ write_set_tcb_field_ulp(tlsp, dst, txq, W_TCB_SND_MAX_RAW,
+ V_TCB_SND_MAX_RAW(M_TCB_SND_MAX_RAW), V_TCB_SND_MAX_RAW(0));
+ dst += roundup2(LEN__SET_TCB_FIELD_ULP, 16);
+
+ if (tp->t_flags & TF_REQ_TSTMP) {
+ write_set_tcb_field_ulp(tlsp, dst, txq, W_TCB_TIMESTAMP_OFFSET,
+ V_TCB_TIMESTAMP_OFFSET(M_TCB_TIMESTAMP_OFFSET),
+ V_TCB_TIMESTAMP_OFFSET(tp->ts_offset >> 28));
+ dst += roundup2(LEN__SET_TCB_FIELD_ULP, 16);
+ }
+
+ KASSERT(dst - (char *)wr == len, ("%s: length mismatch", __func__));
+
+ items[0] = m;
+ error = mp_ring_enqueue(txq->r, items, 1, 1);
+ if (error)
+ m_free(m);
+ return (error);
+}
+
+int
+cxgbe_tls_tag_alloc(struct ifnet *ifp, union if_snd_tag_alloc_params *params,
+ struct m_snd_tag **pt)
+{
+ const struct ktls_session *tls;
+ struct tlspcb *tlsp;
+ struct adapter *sc;
+ struct vi_info *vi;
+ struct inpcb *inp;
+ struct tcpcb *tp;
+ struct sge_txq *txq;
+ int atid, error, keyid;
+
+ tls = params->tls.tls;
+
+ /* Only TLS 1.1 and TLS 1.2 are currently supported. */
+ if (tls->params.tls_vmajor != TLS_MAJOR_VER_ONE ||
+ tls->params.tls_vminor < TLS_MINOR_VER_ONE ||
+ tls->params.tls_vminor > TLS_MINOR_VER_TWO)
+ return (EPROTONOSUPPORT);
+
+ /* Sanity check values in *tls. */
+ switch (tls->params.cipher_algorithm) {
+ case CRYPTO_AES_CBC:
+ /* XXX: Explicitly ignore any provided IV. */
+ switch (tls->params.cipher_key_len) {
+ case 128 / 8:
+ case 192 / 8:
+ case 256 / 8:
+ break;
+ default:
+ return (EINVAL);
+ }
+ switch (tls->params.auth_algorithm) {
+ case CRYPTO_SHA1_HMAC:
+ case CRYPTO_SHA2_256_HMAC:
+ case CRYPTO_SHA2_384_HMAC:
+ break;
+ default:
+ return (EPROTONOSUPPORT);
+ }
+ break;
+ case CRYPTO_AES_NIST_GCM_16:
+ if (tls->params.iv_len != SALT_SIZE)
+ return (EINVAL);
+ switch (tls->params.cipher_key_len) {
+ case 128 / 8:
+ case 192 / 8:
+ case 256 / 8:
+ break;
+ default:
+ return (EINVAL);
+ }
+ break;
+ default:
+ return (EPROTONOSUPPORT);
+ }
+
+ vi = ifp->if_softc;
+ sc = vi->pi->adapter;
+
+ tlsp = alloc_tlspcb(ifp, vi, M_WAITOK);
+
+ atid = alloc_atid(sc, tlsp);
+ if (atid < 0) {
+ error = ENOMEM;
+ goto failed;
+ }
+
+ if (sc->tlst.inline_keys)
+ keyid = -1;
+ else
+ keyid = get_new_keyid(tlsp);
+ if (keyid < 0) {
+ CTR2(KTR_CXGBE, "%s: atid %d using immediate key ctx", __func__,
+ atid);
+ tlsp->inline_key = true;
+ } else {
+ tlsp->tx_key_addr = keyid;
+ CTR3(KTR_CXGBE, "%s: atid %d allocated TX key addr %#x",
+ __func__,
+ atid, tlsp->tx_key_addr);
+ }
+
+ inp = params->tls.inp;
+ INP_RLOCK(inp);
+ if (inp->inp_flags & (INP_TIMEWAIT | INP_DROPPED)) {
+ INP_RUNLOCK(inp);
+ error = ECONNRESET;
+ goto failed;
+ }
+ tlsp->inp = inp;
+
+ tp = inp->inp_ppcb;
+ if (tp->t_flags & TF_REQ_TSTMP) {
+ tlsp->using_timestamps = true;
+ if ((tp->ts_offset & 0xfffffff) != 0) {
+ INP_RUNLOCK(inp);
+ error = EINVAL;
+ goto failed;
+ }
+ } else
+ tlsp->using_timestamps = false;
+
+ error = send_ktls_act_open_req(sc, vi, inp, tlsp, atid);
+ if (error) {
+ INP_RUNLOCK(inp);
+ goto failed;
+ }
+
+ /* Wait for reply to active open. */
+ CTR2(KTR_CXGBE, "%s: atid %d sent CPL_ACT_OPEN_REQ", __func__,
+ atid);
+ while (tlsp->open_pending) {
+ /*
+ * XXX: PCATCH? We would then have to discard the PCB
+ * when the completion CPL arrived.
+ */
+ error = rw_sleep(tlsp, &inp->inp_lock, 0, "t6tlsop", 0);
+ }
+
+ atid = -1;
+ if (tlsp->tid < 0) {
+ INP_RUNLOCK(inp);
+ error = ENOMEM;
+ goto failed;
+ }
+
+ if (inp->inp_flags & (INP_TIMEWAIT | INP_DROPPED)) {
+ INP_RUNLOCK(inp);
+ error = ECONNRESET;
+ goto failed;
+ }
+
+ txq = &sc->sge.txq[vi->first_txq];
+ if (inp->inp_flowtype != M_HASHTYPE_NONE)
+ txq += ((inp->inp_flowid % (vi->ntxq - vi->rsrv_noflowq)) +
+ vi->rsrv_noflowq);
+ tlsp->txq = txq;
+
+ error = ktls_set_tcb_fields(tlsp, tp, txq);
+ INP_RUNLOCK(inp);
+ if (error)
+ goto failed;
+
+ init_ktls_key_params(tlsp, tls);
+
+ error = ktls_setup_keys(tlsp, tls, txq);
+ if (error)
+ goto failed;
+
+ /* The SCMD fields used when encrypting a full TLS record. */
+ tlsp->scmd0.seqno_numivs = htobe32(V_SCMD_SEQ_NO_CTRL(3) |
+ V_SCMD_PROTO_VERSION(tlsp->proto_ver) |
+ V_SCMD_ENC_DEC_CTRL(SCMD_ENCDECCTRL_ENCRYPT) |
+ V_SCMD_CIPH_AUTH_SEQ_CTRL((tlsp->mac_first == 0)) |
+ V_SCMD_CIPH_MODE(tlsp->enc_mode) |
+ V_SCMD_AUTH_MODE(tlsp->auth_mode) |
+ V_SCMD_HMAC_CTRL(tlsp->hmac_ctrl) |
+ V_SCMD_IV_SIZE(tlsp->iv_size) | V_SCMD_NUM_IVS(1));
+
+ tlsp->scmd0.ivgen_hdrlen = V_SCMD_IV_GEN_CTRL(0) |
+ V_SCMD_TLS_FRAG_ENABLE(0);
+ if (tlsp->inline_key)
+ tlsp->scmd0.ivgen_hdrlen |= V_SCMD_KEY_CTX_INLINE(1);
+ tlsp->scmd0.ivgen_hdrlen = htobe32(tlsp->scmd0.ivgen_hdrlen);
+
+ /*
+ * The SCMD fields used when encrypting a partial TLS record
+ * (no trailer and possibly a truncated payload).
+ */
+ tlsp->scmd0_short.seqno_numivs = V_SCMD_SEQ_NO_CTRL(0) |
+ V_SCMD_PROTO_VERSION(SCMD_PROTO_VERSION_GENERIC) |
+ V_SCMD_ENC_DEC_CTRL(SCMD_ENCDECCTRL_ENCRYPT) |
+ V_SCMD_CIPH_AUTH_SEQ_CTRL((tlsp->mac_first == 0)) |
+ V_SCMD_AUTH_MODE(SCMD_AUTH_MODE_NOP) |
+ V_SCMD_HMAC_CTRL(SCMD_HMAC_CTRL_NOP) |
+ V_SCMD_IV_SIZE(AES_BLOCK_LEN / 2) | V_SCMD_NUM_IVS(0);
+ if (tlsp->enc_mode == SCMD_CIPH_MODE_AES_GCM)
+ tlsp->scmd0_short.seqno_numivs |=
+ V_SCMD_CIPH_MODE(SCMD_CIPH_MODE_AES_CTR);
+ else
+ tlsp->scmd0_short.seqno_numivs |=
+ V_SCMD_CIPH_MODE(tlsp->enc_mode);
+ tlsp->scmd0_short.seqno_numivs =
+ htobe32(tlsp->scmd0_short.seqno_numivs);
+
+ tlsp->scmd0_short.ivgen_hdrlen = V_SCMD_IV_GEN_CTRL(0) |
+ V_SCMD_TLS_FRAG_ENABLE(0) |
+ V_SCMD_AADIVDROP(1);
+ if (tlsp->inline_key)
+ tlsp->scmd0_short.ivgen_hdrlen |= V_SCMD_KEY_CTX_INLINE(1);
+
+ TXQ_LOCK(txq);
+ if (tlsp->enc_mode == SCMD_CIPH_MODE_AES_GCM)
+ txq->kern_tls_gcm++;
+ else
+ txq->kern_tls_cbc++;
+ TXQ_UNLOCK(txq);
+ *pt = &tlsp->com.com;
+ return (0);
+
+failed:
+ if (atid >= 0)
+ free_atid(sc, atid);
+ m_snd_tag_rele(&tlsp->com.com);
+ return (error);
+}
+
+static int
+ktls_setup_keys(struct tlspcb *tlsp, const struct ktls_session *tls,
+ struct sge_txq *txq)
+{
+ struct auth_hash *axf;
+ int error, keyid, kwrlen, kctxlen, len;
+ struct tls_key_req *kwr;
+ struct tls_keyctx *kctx;
+ void *items[1], *key;
+ struct tx_keyctx_hdr *khdr;
+ unsigned int ck_size, mk_size, partial_digest_len;
+ struct mbuf *m;
+
+ /*
+ * Store the salt and keys in the key context. For
+ * connections with an inline key, this key context is passed
+ * as immediate data in each work request. For connections
+ * storing the key in DDR, a work request is used to store a
+ * copy of the key context in DDR.
+ */
+ kctx = &tlsp->keyctx;
+ khdr = &kctx->txhdr;
+
+ switch (tlsp->cipher_secret_size) {
+ case 128 / 8:
+ ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_128;
+ break;
+ case 192 / 8:
+ ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_192;
+ break;
+ case 256 / 8:
+ ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_256;
+ break;
+ default:
+ panic("bad key size");
+ }
+ axf = NULL;
+ partial_digest_len = 0;
+ if (tlsp->enc_mode == SCMD_CIPH_MODE_AES_GCM)
+ mk_size = CHCR_KEYCTX_MAC_KEY_SIZE_512;
+ else {
+ switch (tlsp->auth_mode) {
+ case SCMD_AUTH_MODE_SHA1:
+ axf = &auth_hash_hmac_sha1;
+ mk_size = CHCR_KEYCTX_MAC_KEY_SIZE_160;
+ partial_digest_len = SHA1_HASH_LEN;
+ break;
+ case SCMD_AUTH_MODE_SHA256:
+ axf = &auth_hash_hmac_sha2_256;
+ mk_size = CHCR_KEYCTX_MAC_KEY_SIZE_256;
+ partial_digest_len = SHA2_256_HASH_LEN;
+ break;
+ case SCMD_AUTH_MODE_SHA512_384:
+ axf = &auth_hash_hmac_sha2_384;
+ mk_size = CHCR_KEYCTX_MAC_KEY_SIZE_512;
+ partial_digest_len = SHA2_512_HASH_LEN;
+ break;
+ default:
+ panic("bad auth mode");
+ }
+ }
+
+ khdr->ctxlen = (tlsp->tx_key_info_size >> 4);
+ khdr->dualck_to_txvalid = V_TLS_KEYCTX_TX_WR_SALT_PRESENT(1) |
+ V_TLS_KEYCTX_TX_WR_TXCK_SIZE(ck_size) |
+ V_TLS_KEYCTX_TX_WR_TXMK_SIZE(mk_size) |
+ V_TLS_KEYCTX_TX_WR_TXVALID(1);
+ if (tlsp->enc_mode != SCMD_CIPH_MODE_AES_GCM)
+ khdr->dualck_to_txvalid |= V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1);
+ khdr->dualck_to_txvalid = htobe16(khdr->dualck_to_txvalid);
+ key = kctx->keys.edkey;
+ memcpy(key, tls->params.cipher_key, tls->params.cipher_key_len);
+ if (tlsp->enc_mode == SCMD_CIPH_MODE_AES_GCM) {
+ memcpy(khdr->txsalt, tls->params.iv, SALT_SIZE);
+ t4_init_gmac_hash(tls->params.cipher_key,
+ tls->params.cipher_key_len * 8,
+ (char *)key + tls->params.cipher_key_len);
+ } else {
+ t4_init_hmac_digest(axf, partial_digest_len,
+ tls->params.auth_key, tls->params.auth_key_len * 8,
+ (char *)key + tls->params.cipher_key_len);
+ }
+
+ if (tlsp->inline_key)
+ return (0);
+
+ keyid = tlsp->tx_key_addr;
+
+ /* Populate key work request. */
+ kwrlen = sizeof(*kwr);
+ kctxlen = roundup2(sizeof(*kctx), 32);
+ len = kwrlen + kctxlen;
+
+ m = alloc_wr_mbuf(len, M_NOWAIT);
+ if (m == NULL) {
+ CTR2(KTR_CXGBE, "%s: tid %d failed to alloc WR mbuf", __func__,
+ tlsp->tid);
+ return (ENOMEM);
*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
More information about the svn-src-head
mailing list