svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom
John Baldwin
jhb at FreeBSD.org
Tue Mar 13 23:05:53 UTC 2018
Author: jhb
Date: Tue Mar 13 23:05:51 2018
New Revision: 330884
URL: https://svnweb.freebsd.org/changeset/base/330884
Log:
Support for TLS offload of TOE connections on T6 adapters.
The TOE engine in Chelsio T6 adapters supports offloading of TLS
encryption and TCP segmentation for offloaded connections. Sockets
using TLS are required to use a set of custom socket options to upload
RX and TX keys to the NIC and to enable RX processing. Currently
these socket options are implemented as TCP options in the vendor
specific range. A patched OpenSSL library will be made available in a
port / package for use with the TLS TOE support.
TOE sockets can either offload both transmit and reception of TLS
records or just transmit. TLS offload (both RX and TX) is enabled by
setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
enabled on the relevant interface. Transmit offload can be used on
any "normal" or TLS TOE socket by using the custom socket option to
program a transmit key. This permits most TOE sockets to
transparently offload TLS when applications use a patched SSL library
(e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
library). Receive offload can only be used with TOE sockets using the
TLS mode. The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
list of TCP port numbers. Any connection with either a local or
remote port number in that list will be created as a TLS socket rather
than a plain TOE socket. Note that although this sysctl accepts an
arbitrary list of port numbers, the sysctl(8) tool is only able to set
sysctl nodes to a single value. A TLS socket will hang without
receiving data if used by an application that is not using a patched
SSL library. Thus, the tls_rx_ports node should be used with care.
For a server mostly concerned with offloading TLS transmit, this node
is not needed as plain TOE sockets will fall back to software crypto
when using an unpatched SSL library.
New per-interface statistics nodes are added giving counts of TLS
packets and payload bytes (payload bytes do not include TLS headers or
authentication tags/MACs) offloaded via the TOE engine, e.g.:
dev.cc.0.stats.rx_tls_octets: 149
dev.cc.0.stats.rx_tls_records: 13
dev.cc.0.stats.tx_tls_octets: 26501823
dev.cc.0.stats.tx_tls_records: 1620
TLS transmit work requests are constructed by a new variant of
t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
TLS transmit work requests require a buffer containing IVs. If the
IVs are too large to fit into the work request, a separate buffer is
allocated when constructing a work request. This buffer is associated
with the transmit descriptor and freed when the descriptor is ACKed by
the adapter.
Received TLS frames use two new CPL messages. The first message is a
CPL_TLS_DATA containing the decryped payload of a single TLS record.
The handler places the mbuf containing the received payload on an
mbufq in the TOE pcb. The second message is a CPL_RX_TLS_CMP message
which includes a copy of the TLS header and indicates if there were
any errors. The handler for this message places the TLS header into
the socket buffer followed by the saved mbuf with the payload data.
Both of these handlers are contained in tom/t4_tls.c.
A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
including send_rx_credits(), a new send_rx_modulate(), and
t4_close_conn().
TLS keys for both transmit and receive are stored in onboard memory
in the NIC in the "TLS keys" memory region.
In some cases a TLS socket can hang with pending data available in the
NIC that is not delivered to the host. As a workaround, TLS sockets
are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
any data is read from a TLS socket. In addition, a fallback timer will
periodically send CPL_RX_DATA_ACK messages to the NIC for connections
that are still in the handshake phase. Once the connection has
finished the handshake and programmed RX keys via the socket option,
the timer is stopped.
A new function select_ulp_mode() is used to determine what sub-mode a
given TOE socket should use (plain TOE, DDP, or TLS). The existing
set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
handles initialization of TLS-specific state when necessary in
addition to DDP-specific state.
Since TLS sockets do not receive individual TCP segments but always
receive full TLS records, they can receive more data than is available
in the current window (e.g. if a 16k TLS record is received but the
socket buffer is itself 16k). To cope with this, just drop the window
to 0 when this happens, but track the overage and "eat" the overage as
it is read from the socket buffer not opening the window (or adding
rx_credits) for the overage bytes.
Reviewed by: np (earlier version)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D14529
Added:
head/sys/dev/cxgbe/tom/t4_tls.c (contents, props changed)
head/sys/dev/cxgbe/tom/t4_tls.h (contents, props changed)
Modified:
head/sys/dev/cxgbe/adapter.h
head/sys/dev/cxgbe/firmware/t6fw_cfg.txt
head/sys/dev/cxgbe/offload.h
head/sys/dev/cxgbe/t4_main.c
head/sys/dev/cxgbe/tom/t4_connect.c
head/sys/dev/cxgbe/tom/t4_cpl_io.c
head/sys/dev/cxgbe/tom/t4_listen.c
head/sys/dev/cxgbe/tom/t4_tom.c
head/sys/dev/cxgbe/tom/t4_tom.h
head/sys/modules/cxgbe/tom/Makefile
Modified: head/sys/dev/cxgbe/adapter.h
==============================================================================
--- head/sys/dev/cxgbe/adapter.h Tue Mar 13 22:54:29 2018 (r330883)
+++ head/sys/dev/cxgbe/adapter.h Tue Mar 13 23:05:51 2018 (r330884)
@@ -297,6 +297,10 @@ struct port_info {
struct port_stats stats;
u_int tnl_cong_drops;
u_int tx_parse_error;
+ u_long tx_tls_records;
+ u_long tx_tls_octets;
+ u_long rx_tls_records;
+ u_long rx_tls_octets;
struct callout tick;
};
Modified: head/sys/dev/cxgbe/firmware/t6fw_cfg.txt
==============================================================================
--- head/sys/dev/cxgbe/firmware/t6fw_cfg.txt Tue Mar 13 22:54:29 2018 (r330883)
+++ head/sys/dev/cxgbe/firmware/t6fw_cfg.txt Tue Mar 13 23:05:51 2018 (r330884)
@@ -163,10 +163,12 @@
nserver = 512
nhpfilter = 0
nhash = 16384
- protocol = ofld, rddp, rdmac, iscsi_initiator_pdu, iscsi_target_pdu, iscsi_t10dif, crypto_lookaside
+ protocol = ofld, rddp, rdmac, iscsi_initiator_pdu, iscsi_target_pdu, iscsi_t10dif, tlskeys, crypto_lookaside
tp_l2t = 4096
tp_ddp = 2
tp_ddp_iscsi = 2
+ tp_tls_key = 3
+ tp_tls_mxrxsize = 17408 # 16384 + 1024, governs max rx data, pm max xfer len, rx coalesce sizes
tp_stag = 2
tp_pbl = 5
tp_rq = 7
@@ -273,7 +275,7 @@
[fini]
version = 0x1
- checksum = 0x7191019f
+ checksum = 0x9e8952d2
#
# $FreeBSD$
#
Modified: head/sys/dev/cxgbe/offload.h
==============================================================================
--- head/sys/dev/cxgbe/offload.h Tue Mar 13 22:54:29 2018 (r330883)
+++ head/sys/dev/cxgbe/offload.h Tue Mar 13 23:05:51 2018 (r330884)
@@ -151,6 +151,9 @@ struct tom_tunables {
int sndbuf;
int ddp;
int rx_coalesce;
+ int tls;
+ int *tls_rx_ports;
+ int num_tls_rx_ports;
int tx_align;
int tx_zcopy;
};
Modified: head/sys/dev/cxgbe/t4_main.c
==============================================================================
--- head/sys/dev/cxgbe/t4_main.c Tue Mar 13 22:54:29 2018 (r330883)
+++ head/sys/dev/cxgbe/t4_main.c Tue Mar 13 23:05:51 2018 (r330884)
@@ -591,6 +591,7 @@ static int sysctl_wcwr_stats(SYSCTL_HANDLER_ARGS);
static int sysctl_tc_params(SYSCTL_HANDLER_ARGS);
#endif
#ifdef TCP_OFFLOAD
+static int sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS);
static int sysctl_tp_tick(SYSCTL_HANDLER_ARGS);
static int sysctl_tp_dack_timer(SYSCTL_HANDLER_ARGS);
static int sysctl_tp_timer(SYSCTL_HANDLER_ARGS);
@@ -1390,6 +1391,7 @@ t4_detach_common(device_t dev)
free(sc->sge.iqmap, M_CXGBE);
free(sc->sge.eqmap, M_CXGBE);
free(sc->tids.ftid_tab, M_CXGBE);
+ free(sc->tt.tls_rx_ports, M_CXGBE);
t4_destroy_dma_tag(sc);
if (mtx_initialized(&sc->sc_lock)) {
sx_xlock(&t4_list_lock);
@@ -5433,6 +5435,14 @@ t4_sysctls(struct adapter *sc)
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "rx_coalesce",
CTLFLAG_RW, &sc->tt.rx_coalesce, 0, "receive coalescing");
+ sc->tt.tls = 0;
+ SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tls", CTLFLAG_RW,
+ &sc->tt.tls, 0, "Inline TLS allowed");
+
+ SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "tls_rx_ports",
+ CTLTYPE_INT | CTLFLAG_RW, sc, 0, sysctl_tls_rx_ports,
+ "I", "TCP ports that use inline TLS+TOE RX");
+
sc->tt.tx_align = 1;
SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_align",
CTLFLAG_RW, &sc->tt.tx_align, 0, "chop and align payload");
@@ -5836,6 +5846,19 @@ cxgbe_sysctls(struct port_info *pi)
"# of buffer-group 3 truncated packets");
#undef SYSCTL_ADD_T4_PORTSTAT
+
+ SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_records",
+ CTLFLAG_RD, &pi->tx_tls_records,
+ "# of TLS records transmitted");
+ SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_octets",
+ CTLFLAG_RD, &pi->tx_tls_octets,
+ "# of payload octets in transmitted TLS records");
+ SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_records",
+ CTLFLAG_RD, &pi->rx_tls_records,
+ "# of TLS records received");
+ SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_octets",
+ CTLFLAG_RD, &pi->rx_tls_octets,
+ "# of payload octets in received TLS records");
}
static int
@@ -8257,6 +8280,68 @@ done:
#endif
#ifdef TCP_OFFLOAD
+static int
+sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS)
+{
+ struct adapter *sc = arg1;
+ int *old_ports, *new_ports;
+ int i, new_count, rc;
+
+ if (req->newptr == NULL && req->oldptr == NULL)
+ return (SYSCTL_OUT(req, NULL, imax(sc->tt.num_tls_rx_ports, 1) *
+ sizeof(sc->tt.tls_rx_ports[0])));
+
+ rc = begin_synchronized_op(sc, NULL, SLEEP_OK | INTR_OK, "t4tlsrx");
+ if (rc)
+ return (rc);
+
+ if (sc->tt.num_tls_rx_ports == 0) {
+ i = -1;
+ rc = SYSCTL_OUT(req, &i, sizeof(i));
+ } else
+ rc = SYSCTL_OUT(req, sc->tt.tls_rx_ports,
+ sc->tt.num_tls_rx_ports * sizeof(sc->tt.tls_rx_ports[0]));
+ if (rc == 0 && req->newptr != NULL) {
+ new_count = req->newlen / sizeof(new_ports[0]);
+ new_ports = malloc(new_count * sizeof(new_ports[0]), M_CXGBE,
+ M_WAITOK);
+ rc = SYSCTL_IN(req, new_ports, new_count *
+ sizeof(new_ports[0]));
+ if (rc)
+ goto err;
+
+ /* Allow setting to a single '-1' to clear the list. */
+ if (new_count == 1 && new_ports[0] == -1) {
+ ADAPTER_LOCK(sc);
+ old_ports = sc->tt.tls_rx_ports;
+ sc->tt.tls_rx_ports = NULL;
+ sc->tt.num_tls_rx_ports = 0;
+ ADAPTER_UNLOCK(sc);
+ free(old_ports, M_CXGBE);
+ } else {
+ for (i = 0; i < new_count; i++) {
+ if (new_ports[i] < 1 ||
+ new_ports[i] > IPPORT_MAX) {
+ rc = EINVAL;
+ goto err;
+ }
+ }
+
+ ADAPTER_LOCK(sc);
+ old_ports = sc->tt.tls_rx_ports;
+ sc->tt.tls_rx_ports = new_ports;
+ sc->tt.num_tls_rx_ports = new_count;
+ ADAPTER_UNLOCK(sc);
+ free(old_ports, M_CXGBE);
+ new_ports = NULL;
+ }
+ err:
+ free(new_ports, M_CXGBE);
+ }
+ end_synchronized_op(sc, 0);
+ return (rc);
+}
+
static void
unit_conv(char *buf, size_t len, u_int val, u_int factor)
{
Modified: head/sys/dev/cxgbe/tom/t4_connect.c
==============================================================================
--- head/sys/dev/cxgbe/tom/t4_connect.c Tue Mar 13 22:54:29 2018 (r330883)
+++ head/sys/dev/cxgbe/tom/t4_connect.c Tue Mar 13 23:05:51 2018 (r330884)
@@ -142,6 +142,10 @@ do_act_establish(struct sge_iq *iq, const struct rss_h
}
make_established(toep, cpl->snd_isn, cpl->rcv_isn, cpl->tcp_opt);
+
+ if (toep->ulp_mode == ULP_MODE_TLS)
+ tls_establish(toep);
+
done:
INP_WUNLOCK(inp);
CURVNET_RESTORE();
@@ -268,6 +272,11 @@ calc_opt2a(struct socket *so, struct toepcb *toep)
if (toep->ulp_mode == ULP_MODE_TCPDDP)
opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
#endif
+ if (toep->ulp_mode == ULP_MODE_TLS) {
+ opt2 |= F_RX_FC_VALID;
+ opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
+ opt2 |= F_RX_FC_DISABLE;
+ }
return (htobe32(opt2));
}
@@ -378,10 +387,7 @@ t4_connect(struct toedev *tod, struct socket *so, stru
DONT_OFFLOAD_ACTIVE_OPEN(ENOMEM);
toep->vnet = so->so_vnet;
- if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0)
- set_tcpddp_ulp_mode(toep);
- else
- toep->ulp_mode = ULP_MODE_NONE;
+ set_ulp_mode(toep, select_ulp_mode(so, sc));
SOCKBUF_LOCK(&so->so_rcv);
/* opt0 rcv_bufsiz initially, assumes its normal meaning later */
toep->rx_credits = min(select_rcv_wnd(so) >> 10, M_RCV_BUFSIZ);
Modified: head/sys/dev/cxgbe/tom/t4_cpl_io.c
==============================================================================
--- head/sys/dev/cxgbe/tom/t4_cpl_io.c Tue Mar 13 22:54:29 2018 (r330883)
+++ head/sys/dev/cxgbe/tom/t4_cpl_io.c Tue Mar 13 23:05:51 2018 (r330884)
@@ -73,9 +73,6 @@ __FBSDID("$FreeBSD$");
#include "tom/t4_tom_l2t.h"
#include "tom/t4_tom.h"
-#define IS_AIOTX_MBUF(m) \
- ((m)->m_flags & M_EXT && (m)->m_ext.ext_flags & EXT_FLAG_AIOTX)
-
static void t4_aiotx_cancel(struct kaiocb *job);
static void t4_aiotx_queue_toep(struct toepcb *toep);
@@ -106,7 +103,7 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
{
struct wrqe *wr;
struct fw_flowc_wr *flowc;
- unsigned int nparams = ftxp ? 8 : 6, flowclen;
+ unsigned int nparams, flowclen, paramidx;
struct vi_info *vi = toep->vi;
struct port_info *pi = vi->pi;
struct adapter *sc = pi->adapter;
@@ -116,6 +113,15 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
KASSERT(!(toep->flags & TPF_FLOWC_WR_SENT),
("%s: flowc for tid %u sent already", __func__, toep->tid));
+ if (ftxp != NULL)
+ nparams = 8;
+ else
+ nparams = 6;
+ if (toep->ulp_mode == ULP_MODE_TLS)
+ nparams++;
+ if (toep->tls.fcplenmax != 0)
+ nparams++;
+
flowclen = sizeof(*flowc) + nparams * sizeof(struct fw_flowc_mnemval);
wr = alloc_wrqe(roundup2(flowclen, 16), toep->ofld_txq);
@@ -131,39 +137,45 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
flowc->flowid_len16 = htonl(V_FW_WR_LEN16(howmany(flowclen, 16)) |
V_FW_WR_FLOWID(toep->tid));
- flowc->mnemval[0].mnemonic = FW_FLOWC_MNEM_PFNVFN;
- flowc->mnemval[0].val = htobe32(pfvf);
- flowc->mnemval[1].mnemonic = FW_FLOWC_MNEM_CH;
- flowc->mnemval[1].val = htobe32(pi->tx_chan);
- flowc->mnemval[2].mnemonic = FW_FLOWC_MNEM_PORT;
- flowc->mnemval[2].val = htobe32(pi->tx_chan);
- flowc->mnemval[3].mnemonic = FW_FLOWC_MNEM_IQID;
- flowc->mnemval[3].val = htobe32(toep->ofld_rxq->iq.abs_id);
+#define FLOWC_PARAM(__m, __v) \
+ do { \
+ flowc->mnemval[paramidx].mnemonic = FW_FLOWC_MNEM_##__m; \
+ flowc->mnemval[paramidx].val = htobe32(__v); \
+ paramidx++; \
+ } while (0)
+
+ paramidx = 0;
+
+ FLOWC_PARAM(PFNVFN, pfvf);
+ FLOWC_PARAM(CH, pi->tx_chan);
+ FLOWC_PARAM(PORT, pi->tx_chan);
+ FLOWC_PARAM(IQID, toep->ofld_rxq->iq.abs_id);
if (ftxp) {
uint32_t sndbuf = min(ftxp->snd_space, sc->tt.sndbuf);
- flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDNXT;
- flowc->mnemval[4].val = htobe32(ftxp->snd_nxt);
- flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_RCVNXT;
- flowc->mnemval[5].val = htobe32(ftxp->rcv_nxt);
- flowc->mnemval[6].mnemonic = FW_FLOWC_MNEM_SNDBUF;
- flowc->mnemval[6].val = htobe32(sndbuf);
- flowc->mnemval[7].mnemonic = FW_FLOWC_MNEM_MSS;
- flowc->mnemval[7].val = htobe32(ftxp->mss);
+ FLOWC_PARAM(SNDNXT, ftxp->snd_nxt);
+ FLOWC_PARAM(RCVNXT, ftxp->rcv_nxt);
+ FLOWC_PARAM(SNDBUF, sndbuf);
+ FLOWC_PARAM(MSS, ftxp->mss);
CTR6(KTR_CXGBE,
"%s: tid %u, mss %u, sndbuf %u, snd_nxt 0x%x, rcv_nxt 0x%x",
__func__, toep->tid, ftxp->mss, sndbuf, ftxp->snd_nxt,
ftxp->rcv_nxt);
} else {
- flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDBUF;
- flowc->mnemval[4].val = htobe32(512);
- flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_MSS;
- flowc->mnemval[5].val = htobe32(512);
+ FLOWC_PARAM(SNDBUF, 512);
+ FLOWC_PARAM(MSS, 512);
CTR2(KTR_CXGBE, "%s: tid %u", __func__, toep->tid);
}
+ if (toep->ulp_mode == ULP_MODE_TLS)
+ FLOWC_PARAM(ULP_MODE, toep->ulp_mode);
+ if (toep->tls.fcplenmax != 0)
+ FLOWC_PARAM(TXDATAPLEN_MAX, toep->tls.fcplenmax);
+#undef FLOWC_PARAM
+ KASSERT(paramidx == nparams, ("nparams mismatch"));
+
txsd->tx_credits = howmany(flowclen, 16);
txsd->plen = 0;
KASSERT(toep->tx_credits >= txsd->tx_credits && toep->txsd_avail > 0,
@@ -421,7 +433,7 @@ make_established(struct toepcb *toep, uint32_t snd_isn
soisconnected(so);
}
-static int
+int
send_rx_credits(struct adapter *sc, struct toepcb *toep, int credits)
{
struct wrqe *wr;
@@ -443,6 +455,23 @@ send_rx_credits(struct adapter *sc, struct toepcb *toe
}
void
+send_rx_modulate(struct adapter *sc, struct toepcb *toep)
+{
+ struct wrqe *wr;
+ struct cpl_rx_data_ack *req;
+
+ wr = alloc_wrqe(sizeof(*req), toep->ctrlq);
+ if (wr == NULL)
+ return;
+ req = wrtod(wr);
+
+ INIT_TP_WR_MIT_CPL(req, CPL_RX_DATA_ACK, toep->tid);
+ req->credit_dack = htobe32(F_RX_MODULATE_RX);
+
+ t4_wrq_tx(sc, wr);
+}
+
+void
t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
{
struct adapter *sc = tod->tod_softc;
@@ -459,8 +488,18 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
("%s: sb %p has more data (%d) than last time (%d).",
__func__, sb, sbused(sb), toep->sb_cc));
- toep->rx_credits += toep->sb_cc - sbused(sb);
+ credits = toep->sb_cc - sbused(sb);
toep->sb_cc = sbused(sb);
+ if (toep->ulp_mode == ULP_MODE_TLS) {
+ if (toep->tls.rcv_over >= credits) {
+ toep->tls.rcv_over -= credits;
+ credits = 0;
+ } else {
+ credits -= toep->tls.rcv_over;
+ toep->tls.rcv_over = 0;
+ }
+ }
+ toep->rx_credits += credits;
if (toep->rx_credits > 0 &&
(tp->rcv_wnd <= 32 * 1024 || toep->rx_credits >= 64 * 1024 ||
@@ -471,7 +510,8 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
toep->rx_credits -= credits;
tp->rcv_wnd += credits;
tp->rcv_adv += credits;
- }
+ } else if (toep->flags & TPF_FORCE_CREDITS)
+ send_rx_modulate(sc, toep);
}
void
@@ -489,8 +529,8 @@ t4_rcvd(struct toedev *tod, struct tcpcb *tp)
/*
* Close a connection by sending a CPL_CLOSE_CON_REQ message.
*/
-static int
-close_conn(struct adapter *sc, struct toepcb *toep)
+int
+t4_close_conn(struct adapter *sc, struct toepcb *toep)
{
struct wrqe *wr;
struct cpl_close_con_req *req;
@@ -691,6 +731,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep
KASSERT(toep->ulp_mode == ULP_MODE_NONE ||
toep->ulp_mode == ULP_MODE_TCPDDP ||
+ toep->ulp_mode == ULP_MODE_TLS ||
toep->ulp_mode == ULP_MODE_RDMA,
("%s: ulp_mode %u for toep %p", __func__, toep->ulp_mode, toep));
@@ -905,7 +946,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep
/* Send a FIN if requested, but only if there's no more data to send */
if (m == NULL && toep->flags & TPF_SEND_FIN)
- close_conn(sc, toep);
+ t4_close_conn(sc, toep);
}
static inline void
@@ -1097,7 +1138,7 @@ t4_push_pdus(struct adapter *sc, struct toepcb *toep,
/* Send a FIN if requested, but only if there are no more PDUs to send */
if (mbufq_first(pduq) == NULL && toep->flags & TPF_SEND_FIN)
- close_conn(sc, toep);
+ t4_close_conn(sc, toep);
}
int
@@ -1116,6 +1157,8 @@ t4_tod_output(struct toedev *tod, struct tcpcb *tp)
if (toep->ulp_mode == ULP_MODE_ISCSI)
t4_push_pdus(sc, toep, 0);
+ else if (tls_tx_key(toep))
+ t4_push_tls_records(sc, toep, 0);
else
t4_push_frames(sc, toep, 0);
@@ -1140,6 +1183,8 @@ t4_send_fin(struct toedev *tod, struct tcpcb *tp)
if (tp->t_state >= TCPS_ESTABLISHED) {
if (toep->ulp_mode == ULP_MODE_ISCSI)
t4_push_pdus(sc, toep, 0);
+ else if (tls_tx_key(toep))
+ t4_push_tls_records(sc, toep, 0);
else
t4_push_frames(sc, toep, 0);
}
@@ -1772,6 +1817,10 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header
credits -= txsd->tx_credits;
toep->tx_credits += txsd->tx_credits;
plen += txsd->plen;
+ if (txsd->iv_buffer) {
+ free(txsd->iv_buffer, M_CXGBE);
+ txsd->iv_buffer = NULL;
+ }
txsd++;
toep->txsd_avail++;
KASSERT(toep->txsd_avail <= toep->txsd_total,
@@ -1797,6 +1846,8 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header
CURVNET_SET(toep->vnet);
if (toep->ulp_mode == ULP_MODE_ISCSI)
t4_push_pdus(sc, toep, plen);
+ else if (tls_tx_key(toep))
+ t4_push_tls_records(sc, toep, plen);
else
t4_push_frames(sc, toep, plen);
CURVNET_RESTORE();
@@ -1826,6 +1877,12 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header
tid, plen);
#endif
sbdrop_locked(sb, plen);
+ if (tls_tx_key(toep)) {
+ struct tls_ofld_info *tls_ofld = &toep->tls;
+
+ MPASS(tls_ofld->sb_off >= plen);
+ tls_ofld->sb_off -= plen;
+ }
if (!TAILQ_EMPTY(&toep->aiotx_jobq))
t4_aiotx_queue_toep(toep);
sowwakeup_locked(so); /* unlocks so_snd */
@@ -2298,6 +2355,9 @@ t4_aio_queue_aiotx(struct socket *so, struct kaiocb *j
return (EOPNOTSUPP);
if (!sc->tt.tx_zcopy)
+ return (EOPNOTSUPP);
+
+ if (is_tls_offload(toep) || tls_tx_key(toep))
return (EOPNOTSUPP);
SOCKBUF_LOCK(&so->so_snd);
Modified: head/sys/dev/cxgbe/tom/t4_listen.c
==============================================================================
--- head/sys/dev/cxgbe/tom/t4_listen.c Tue Mar 13 22:54:29 2018 (r330883)
+++ head/sys/dev/cxgbe/tom/t4_listen.c Tue Mar 13 23:05:51 2018 (r330884)
@@ -1056,6 +1056,11 @@ calc_opt2p(struct adapter *sc, struct port_info *pi, i
if (ulp_mode == ULP_MODE_TCPDDP)
opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
#endif
+ if (ulp_mode == ULP_MODE_TLS) {
+ opt2 |= F_RX_FC_VALID;
+ opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
+ opt2 |= F_RX_FC_DISABLE;
+ }
return htobe32(opt2);
}
@@ -1347,11 +1352,15 @@ found:
INIT_TP_WR_MIT_CPL(rpl5, CPL_PASS_ACCEPT_RPL, tid);
}
- if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0) {
- ulp_mode = ULP_MODE_TCPDDP;
+ ulp_mode = select_ulp_mode(so, sc);
+ switch (ulp_mode) {
+ case ULP_MODE_TCPDDP:
synqe->flags |= TPF_SYNQE_TCPDDP;
- } else
- ulp_mode = ULP_MODE_NONE;
+ break;
+ case ULP_MODE_TLS:
+ synqe->flags |= TPF_SYNQE_TLS;
+ break;
+ }
rpl->opt0 = calc_opt0(so, vi, e, mtu_idx, rscale, rx_credits, ulp_mode);
rpl->opt2 = calc_opt2p(sc, pi, rxqid, &cpl->tcpopt, &th, ulp_mode);
@@ -1407,8 +1416,8 @@ found:
REJECT_PASS_ACCEPT();
}
- CTR5(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK",
- __func__, stid, tid, lctx, synqe);
+ CTR6(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK mode %d",
+ __func__, stid, tid, lctx, synqe, ulp_mode);
INP_WLOCK(inp);
synqe->flags |= TPF_SYNQE_HAS_L2TE;
@@ -1557,9 +1566,11 @@ reset:
toep->tid = tid;
toep->l2te = &sc->l2t->l2tab[synqe->l2e_idx];
if (synqe->flags & TPF_SYNQE_TCPDDP)
- set_tcpddp_ulp_mode(toep);
+ set_ulp_mode(toep, ULP_MODE_TCPDDP);
+ else if (synqe->flags & TPF_SYNQE_TLS)
+ set_ulp_mode(toep, ULP_MODE_TLS);
else
- toep->ulp_mode = ULP_MODE_NONE;
+ set_ulp_mode(toep, ULP_MODE_NONE);
/* opt0 rcv_bufsiz initially, assumes its normal meaning later */
toep->rx_credits = synqe->rcv_bufsize;
Added: head/sys/dev/cxgbe/tom/t4_tls.c
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ head/sys/dev/cxgbe/tom/t4_tls.c Tue Mar 13 23:05:51 2018 (r330884)
@@ -0,0 +1,1642 @@
+/*-
+ * SPDX-License-Identifier: BSD-2-Clause-FreeBSD
+ *
+ * Copyright (c) 2017-2018 Chelsio Communications, Inc.
+ * All rights reserved.
+ * Written by: John Baldwin <jhb at FreeBSD.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "opt_inet.h"
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <sys/param.h>
+#include <sys/sglist.h>
+#include <sys/socket.h>
+#include <sys/socketvar.h>
+#include <sys/systm.h>
+#include <netinet/in.h>
+#include <netinet/in_pcb.h>
+#include <netinet/tcp_var.h>
+#include <netinet/toecore.h>
+
+#ifdef TCP_OFFLOAD
+#include "common/common.h"
+#include "common/t4_tcb.h"
+#include "tom/t4_tom_l2t.h"
+#include "tom/t4_tom.h"
+
+/*
+ * The TCP sequence number of a CPL_TLS_DATA mbuf is saved here while
+ * the mbuf is in the ulp_pdu_reclaimq.
+ */
+#define tls_tcp_seq PH_loc.thirtytwo[0]
+
+/*
+ * Handshake lock used for the handshake timer. Having a global lock
+ * is perhaps not ideal, but it avoids having to use callout_drain()
+ * in tls_uninit_toep() which can't block. Also, the timer shouldn't
+ * actually fire for most connections.
+ */
+static struct mtx tls_handshake_lock;
+
+static void
+t4_set_tls_tcb_field(struct toepcb *toep, uint16_t word, uint64_t mask,
+ uint64_t val)
+{
+ struct adapter *sc = td_adapter(toep->td);
+
+ t4_set_tcb_field(sc, toep->ctrlq, toep->tid, word, mask, val, 0, 0,
+ toep->ofld_rxq->iq.abs_id);
+}
+
+/* TLS and DTLS common routines */
+int
+tls_tx_key(struct toepcb *toep)
+{
+ struct tls_ofld_info *tls_ofld = &toep->tls;
+
+ return (tls_ofld->tx_key_addr >= 0);
+}
+
+int
+tls_rx_key(struct toepcb *toep)
+{
+ struct tls_ofld_info *tls_ofld = &toep->tls;
+
+ return (tls_ofld->rx_key_addr >= 0);
+}
+
+static int
+key_size(struct toepcb *toep)
+{
+ struct tls_ofld_info *tls_ofld = &toep->tls;
+
+ return ((tls_ofld->key_location == TLS_SFO_WR_CONTEXTLOC_IMMEDIATE) ?
+ tls_ofld->k_ctx.tx_key_info_size : KEY_IN_DDR_SIZE);
+}
+
+/* Set TLS Key-Id in TCB */
+static void
+t4_set_tls_keyid(struct toepcb *toep, unsigned int key_id)
+{
+
+ t4_set_tls_tcb_field(toep, W_TCB_RX_TLS_KEY_TAG,
+ V_TCB_RX_TLS_KEY_TAG(M_TCB_RX_TLS_BUF_TAG),
+ V_TCB_RX_TLS_KEY_TAG(key_id));
+}
+
+/* Clear TF_RX_QUIESCE to re-enable receive. */
+static void
+t4_clear_rx_quiesce(struct toepcb *toep)
+{
+
+ t4_set_tls_tcb_field(toep, W_TCB_T_FLAGS, V_TF_RX_QUIESCE(1), 0);
+}
+
+static void
+tls_clr_ofld_mode(struct toepcb *toep)
+{
+
+ tls_stop_handshake_timer(toep);
+
+ /* Operate in PDU extraction mode only. */
+ t4_set_tls_tcb_field(toep, W_TCB_ULP_RAW,
+ V_TCB_ULP_RAW(M_TCB_ULP_RAW),
+ V_TCB_ULP_RAW(V_TF_TLS_ENABLE(1)));
+ t4_clear_rx_quiesce(toep);
+}
+
+static void
+tls_clr_quiesce(struct toepcb *toep)
+{
+
+ tls_stop_handshake_timer(toep);
+ t4_clear_rx_quiesce(toep);
+}
+
+/*
+ * Calculate the TLS data expansion size
+ */
+static int
+tls_expansion_size(struct toepcb *toep, int data_len, int full_pdus_only,
+ unsigned short *pdus_per_ulp)
+{
+ struct tls_ofld_info *tls_ofld = &toep->tls;
+ struct tls_scmd *scmd = &tls_ofld->scmd0;
+ int expn_size = 0, frag_count = 0, pad_per_pdu = 0,
+ pad_last_pdu = 0, last_frag_size = 0, max_frag_size = 0;
+ int exp_per_pdu = 0;
+ int hdr_len = TLS_HEADER_LENGTH;
+
+ do {
+ max_frag_size = tls_ofld->k_ctx.frag_size;
+ if (G_SCMD_CIPH_MODE(scmd->seqno_numivs) ==
+ SCMD_CIPH_MODE_AES_GCM) {
+ frag_count = (data_len / max_frag_size);
+ exp_per_pdu = GCM_TAG_SIZE + AEAD_EXPLICIT_DATA_SIZE +
+ hdr_len;
+ expn_size = frag_count * exp_per_pdu;
+ if (full_pdus_only) {
+ *pdus_per_ulp = data_len / (exp_per_pdu +
+ max_frag_size);
+ if (*pdus_per_ulp > 32)
+ *pdus_per_ulp = 32;
+ else if(!*pdus_per_ulp)
+ *pdus_per_ulp = 1;
+ expn_size = (*pdus_per_ulp) * exp_per_pdu;
+ break;
+ }
+ if ((last_frag_size = data_len % max_frag_size) > 0) {
+ frag_count += 1;
+ expn_size += exp_per_pdu;
+ }
+ break;
+ } else if (G_SCMD_CIPH_MODE(scmd->seqno_numivs) !=
+ SCMD_CIPH_MODE_NOP) {
+ /* Calculate the number of fragments we can make */
+ frag_count = (data_len / max_frag_size);
+ if (frag_count > 0) {
+ pad_per_pdu = (((howmany((max_frag_size +
+ tls_ofld->mac_length),
+ CIPHER_BLOCK_SIZE)) *
+ CIPHER_BLOCK_SIZE) -
+ (max_frag_size +
+ tls_ofld->mac_length));
+ if (!pad_per_pdu)
+ pad_per_pdu = CIPHER_BLOCK_SIZE;
+ exp_per_pdu = pad_per_pdu +
+ tls_ofld->mac_length +
+ hdr_len + CIPHER_BLOCK_SIZE;
+ expn_size = frag_count * exp_per_pdu;
+ }
+ if (full_pdus_only) {
+ *pdus_per_ulp = data_len / (exp_per_pdu +
+ max_frag_size);
+ if (*pdus_per_ulp > 32)
+ *pdus_per_ulp = 32;
+ else if (!*pdus_per_ulp)
+ *pdus_per_ulp = 1;
+ expn_size = (*pdus_per_ulp) * exp_per_pdu;
+ break;
+ }
+ /* Consider the last fragment */
+ if ((last_frag_size = data_len % max_frag_size) > 0) {
+ pad_last_pdu = (((howmany((last_frag_size +
+ tls_ofld->mac_length),
+ CIPHER_BLOCK_SIZE)) *
+ CIPHER_BLOCK_SIZE) -
+ (last_frag_size +
+ tls_ofld->mac_length));
+ if (!pad_last_pdu)
+ pad_last_pdu = CIPHER_BLOCK_SIZE;
+ expn_size += (pad_last_pdu +
+ tls_ofld->mac_length + hdr_len +
+ CIPHER_BLOCK_SIZE);
+ }
+ }
+ } while (0);
+
+ return (expn_size);
+}
+
+/* Copy Key to WR */
+static void
+tls_copy_tx_key(struct toepcb *toep, void *dst)
+{
+ struct tls_ofld_info *tls_ofld = &toep->tls;
+ struct ulptx_sc_memrd *sc_memrd;
+ struct ulptx_idata *sc;
+
+ if (tls_ofld->k_ctx.tx_key_info_size <= 0)
+ return;
+
+ if (tls_ofld->key_location == TLS_SFO_WR_CONTEXTLOC_DDR) {
+ sc = dst;
+ sc->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_NOOP));
+ sc->len = htobe32(0);
+ sc_memrd = (struct ulptx_sc_memrd *)(sc + 1);
+ sc_memrd->cmd_to_len = htobe32(V_ULPTX_CMD(ULP_TX_SC_MEMRD) |
+ V_ULP_TX_SC_MORE(1) |
+ V_ULPTX_LEN16(tls_ofld->k_ctx.tx_key_info_size >> 4));
+ sc_memrd->addr = htobe32(tls_ofld->tx_key_addr >> 5);
+ } else if (tls_ofld->key_location == TLS_SFO_WR_CONTEXTLOC_IMMEDIATE) {
+ memcpy(dst, &tls_ofld->k_ctx.tx,
+ tls_ofld->k_ctx.tx_key_info_size);
+ }
+}
+
+/* TLS/DTLS content type for CPL SFO */
+static inline unsigned char
+tls_content_type(unsigned char content_type)
+{
+ /*
+ * XXX: Shouldn't this map CONTENT_TYPE_APP_DATA to DATA and
+ * default to "CUSTOM" for all other types including
+ * heartbeat?
+ */
+ switch (content_type) {
+ case CONTENT_TYPE_CCS:
+ return CPL_TX_TLS_SFO_TYPE_CCS;
+ case CONTENT_TYPE_ALERT:
+ return CPL_TX_TLS_SFO_TYPE_ALERT;
+ case CONTENT_TYPE_HANDSHAKE:
+ return CPL_TX_TLS_SFO_TYPE_HANDSHAKE;
+ case CONTENT_TYPE_HEARTBEAT:
+ return CPL_TX_TLS_SFO_TYPE_HEARTBEAT;
+ }
+ return CPL_TX_TLS_SFO_TYPE_DATA;
+}
+
+static unsigned char
+get_cipher_key_size(unsigned int ck_size)
+{
+ switch (ck_size) {
+ case AES_NOP: /* NOP */
+ return 15;
+ case AES_128: /* AES128 */
+ return CH_CK_SIZE_128;
+ case AES_192: /* AES192 */
+ return CH_CK_SIZE_192;
+ case AES_256: /* AES256 */
+ return CH_CK_SIZE_256;
+ default:
+ return CH_CK_SIZE_256;
+ }
+}
+
+static unsigned char
+get_mac_key_size(unsigned int mk_size)
+{
+ switch (mk_size) {
+ case SHA_NOP: /* NOP */
+ return CH_MK_SIZE_128;
+ case SHA_GHASH: /* GHASH */
+ case SHA_512: /* SHA512 */
+ return CH_MK_SIZE_512;
+ case SHA_224: /* SHA2-224 */
+ return CH_MK_SIZE_192;
+ case SHA_256: /* SHA2-256*/
+ return CH_MK_SIZE_256;
+ case SHA_384: /* SHA384 */
+ return CH_MK_SIZE_512;
+ case SHA1: /* SHA1 */
+ default:
+ return CH_MK_SIZE_160;
+ }
+}
+
+static unsigned int
+get_proto_ver(int proto_ver)
+{
+ switch (proto_ver) {
+ case TLS1_2_VERSION:
+ return TLS_1_2_VERSION;
+ case TLS1_1_VERSION:
+ return TLS_1_1_VERSION;
+ case DTLS1_2_VERSION:
+ return DTLS_1_2_VERSION;
+ default:
+ return TLS_VERSION_MAX;
+ }
+}
+
+static void
+tls_rxkey_flit1(struct tls_keyctx *kwr, struct tls_key_context *kctx)
+{
+
+ if (kctx->state.enc_mode == CH_EVP_CIPH_GCM_MODE) {
+ kwr->u.rxhdr.ivinsert_to_authinsrt =
+ htobe64(V_TLS_KEYCTX_TX_WR_IVINSERT(6ULL) |
+ V_TLS_KEYCTX_TX_WR_AADSTRTOFST(1ULL) |
+ V_TLS_KEYCTX_TX_WR_AADSTOPOFST(5ULL) |
+ V_TLS_KEYCTX_TX_WR_AUTHSRTOFST(14ULL) |
+ V_TLS_KEYCTX_TX_WR_AUTHSTOPOFST(16ULL) |
+ V_TLS_KEYCTX_TX_WR_CIPHERSRTOFST(14ULL) |
+ V_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST(0ULL) |
+ V_TLS_KEYCTX_TX_WR_AUTHINSRT(16ULL));
+ kwr->u.rxhdr.ivpresent_to_rxmk_size &=
+ ~(V_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(1));
+ kwr->u.rxhdr.authmode_to_rxvalid &=
+ ~(V_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(1));
+ } else {
+ kwr->u.rxhdr.ivinsert_to_authinsrt =
+ htobe64(V_TLS_KEYCTX_TX_WR_IVINSERT(6ULL) |
+ V_TLS_KEYCTX_TX_WR_AADSTRTOFST(1ULL) |
+ V_TLS_KEYCTX_TX_WR_AADSTOPOFST(5ULL) |
+ V_TLS_KEYCTX_TX_WR_AUTHSRTOFST(22ULL) |
+ V_TLS_KEYCTX_TX_WR_AUTHSTOPOFST(0ULL) |
+ V_TLS_KEYCTX_TX_WR_CIPHERSRTOFST(22ULL) |
+ V_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST(0ULL) |
+ V_TLS_KEYCTX_TX_WR_AUTHINSRT(0ULL));
+ }
+}
+
+/* Rx key */
+static void
+prepare_rxkey_wr(struct tls_keyctx *kwr, struct tls_key_context *kctx)
+{
+ unsigned int ck_size = kctx->cipher_secret_size;
+ unsigned int mk_size = kctx->mac_secret_size;
+ int proto_ver = kctx->proto_ver;
+
+ kwr->u.rxhdr.flitcnt_hmacctrl =
+ ((kctx->tx_key_info_size >> 4) << 3) | kctx->hmac_ctrl;
+
+ kwr->u.rxhdr.protover_ciphmode =
+ V_TLS_KEYCTX_TX_WR_PROTOVER(get_proto_ver(proto_ver)) |
+ V_TLS_KEYCTX_TX_WR_CIPHMODE(kctx->state.enc_mode);
+
+ kwr->u.rxhdr.authmode_to_rxvalid =
+ V_TLS_KEYCTX_TX_WR_AUTHMODE(kctx->state.auth_mode) |
+ V_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(1) |
+ V_TLS_KEYCTX_TX_WR_SEQNUMCTRL(3) |
+ V_TLS_KEYCTX_TX_WR_RXVALID(1);
+
+ kwr->u.rxhdr.ivpresent_to_rxmk_size =
+ V_TLS_KEYCTX_TX_WR_IVPRESENT(0) |
+ V_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(1) |
+ V_TLS_KEYCTX_TX_WR_RXCK_SIZE(get_cipher_key_size(ck_size)) |
+ V_TLS_KEYCTX_TX_WR_RXMK_SIZE(get_mac_key_size(mk_size));
+
+ tls_rxkey_flit1(kwr, kctx);
+
+ /* No key reversal for GCM */
+ if (kctx->state.enc_mode != CH_EVP_CIPH_GCM_MODE) {
+ t4_aes_getdeckey(kwr->keys.edkey, kctx->rx.key,
+ (kctx->cipher_secret_size << 3));
+ memcpy(kwr->keys.edkey + kctx->cipher_secret_size,
+ kctx->rx.key + kctx->cipher_secret_size,
+ (IPAD_SIZE + OPAD_SIZE));
+ } else {
+ memcpy(kwr->keys.edkey, kctx->rx.key,
+ (kctx->tx_key_info_size - SALT_SIZE));
+ memcpy(kwr->u.rxhdr.rxsalt, kctx->rx.salt, SALT_SIZE);
+ }
+}
+
+/* Tx key */
+static void
+prepare_txkey_wr(struct tls_keyctx *kwr, struct tls_key_context *kctx)
+{
+ unsigned int ck_size = kctx->cipher_secret_size;
+ unsigned int mk_size = kctx->mac_secret_size;
+
+ kwr->u.txhdr.ctxlen =
+ (kctx->tx_key_info_size >> 4);
+ kwr->u.txhdr.dualck_to_txvalid =
+ V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1) |
+ V_TLS_KEYCTX_TX_WR_SALT_PRESENT(1) |
+ V_TLS_KEYCTX_TX_WR_TXCK_SIZE(get_cipher_key_size(ck_size)) |
+ V_TLS_KEYCTX_TX_WR_TXMK_SIZE(get_mac_key_size(mk_size)) |
+ V_TLS_KEYCTX_TX_WR_TXVALID(1);
+
+ memcpy(kwr->keys.edkey, kctx->tx.key, HDR_KCTX_SIZE);
+ if (kctx->state.enc_mode == CH_EVP_CIPH_GCM_MODE) {
+ memcpy(kwr->u.txhdr.txsalt, kctx->tx.salt, SALT_SIZE);
+ kwr->u.txhdr.dualck_to_txvalid &=
+ ~(V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1));
+ }
+ kwr->u.txhdr.dualck_to_txvalid = htons(kwr->u.txhdr.dualck_to_txvalid);
+}
+
+/* TLS Key memory management */
+int
+tls_init_kmap(struct adapter *sc, struct tom_data *td)
+{
+
+ td->key_map = vmem_create("T4TLS key map", sc->vres.key.start,
+ sc->vres.key.size, 8, 0, M_FIRSTFIT | M_NOWAIT);
+ if (td->key_map == NULL)
+ return (ENOMEM);
+ return (0);
+}
+
+void
+tls_free_kmap(struct tom_data *td)
+{
+
+ if (td->key_map != NULL)
+ vmem_destroy(td->key_map);
+}
+
+static int
+get_new_keyid(struct toepcb *toep, struct tls_key_context *k_ctx)
+{
+ struct tom_data *td = toep->td;
+ vmem_addr_t addr;
+
+ if (vmem_alloc(td->key_map, TLS_KEY_CONTEXT_SZ, M_NOWAIT | M_FIRSTFIT,
+ &addr) != 0)
+ return (-1);
+
+ return (addr);
+}
+
+static void
+free_keyid(struct toepcb *toep, int keyid)
+{
+ struct tom_data *td = toep->td;
+
+ vmem_free(td->key_map, keyid, TLS_KEY_CONTEXT_SZ);
+}
+
*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
More information about the svn-src-head
mailing list