svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom
O. Hartmann
ohartmann at walstatt.org
Wed Mar 14 06:36:54 UTC 2018
On Wed, 14 Mar 2018 06:25:10 +0100
"O. Hartmann" <ohartmann at walstatt.org> wrote:
> On Tue, 13 Mar 2018 23:05:51 +0000 (UTC)
> John Baldwin <jhb at FreeBSD.org> wrote:
>
> > Author: jhb
> > Date: Tue Mar 13 23:05:51 2018
> > New Revision: 330884
> > URL: https://svnweb.freebsd.org/changeset/base/330884
> >
> > Log:
> > Support for TLS offload of TOE connections on T6 adapters.
> >
> > The TOE engine in Chelsio T6 adapters supports offloading of TLS
> > encryption and TCP segmentation for offloaded connections. Sockets
> > using TLS are required to use a set of custom socket options to upload
> > RX and TX keys to the NIC and to enable RX processing. Currently
> > these socket options are implemented as TCP options in the vendor
> > specific range. A patched OpenSSL library will be made available in a
> > port / package for use with the TLS TOE support.
> >
> > TOE sockets can either offload both transmit and reception of TLS
> > records or just transmit. TLS offload (both RX and TX) is enabled by
> > setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
> > enabled on the relevant interface. Transmit offload can be used on
> > any "normal" or TLS TOE socket by using the custom socket option to
> > program a transmit key. This permits most TOE sockets to
> > transparently offload TLS when applications use a patched SSL library
> > (e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
> > library). Receive offload can only be used with TOE sockets using the
> > TLS mode. The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
> > list of TCP port numbers. Any connection with either a local or
> > remote port number in that list will be created as a TLS socket rather
> > than a plain TOE socket. Note that although this sysctl accepts an
> > arbitrary list of port numbers, the sysctl(8) tool is only able to set
> > sysctl nodes to a single value. A TLS socket will hang without
> > receiving data if used by an application that is not using a patched
> > SSL library. Thus, the tls_rx_ports node should be used with care.
> > For a server mostly concerned with offloading TLS transmit, this node
> > is not needed as plain TOE sockets will fall back to software crypto
> > when using an unpatched SSL library.
> >
> > New per-interface statistics nodes are added giving counts of TLS
> > packets and payload bytes (payload bytes do not include TLS headers or
> > authentication tags/MACs) offloaded via the TOE engine, e.g.:
> >
> > dev.cc.0.stats.rx_tls_octets: 149
> > dev.cc.0.stats.rx_tls_records: 13
> > dev.cc.0.stats.tx_tls_octets: 26501823
> > dev.cc.0.stats.tx_tls_records: 1620
> >
> > TLS transmit work requests are constructed by a new variant of
> > t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
> >
> > TLS transmit work requests require a buffer containing IVs. If the
> > IVs are too large to fit into the work request, a separate buffer is
> > allocated when constructing a work request. This buffer is associated
> > with the transmit descriptor and freed when the descriptor is ACKed by
> > the adapter.
> >
> > Received TLS frames use two new CPL messages. The first message is a
> > CPL_TLS_DATA containing the decryped payload of a single TLS record.
> > The handler places the mbuf containing the received payload on an
> > mbufq in the TOE pcb. The second message is a CPL_RX_TLS_CMP message
> > which includes a copy of the TLS header and indicates if there were
> > any errors. The handler for this message places the TLS header into
> > the socket buffer followed by the saved mbuf with the payload data.
> > Both of these handlers are contained in tom/t4_tls.c.
> >
> > A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
> > including send_rx_credits(), a new send_rx_modulate(), and
> > t4_close_conn().
> >
> > TLS keys for both transmit and receive are stored in onboard memory
> > in the NIC in the "TLS keys" memory region.
> >
> > In some cases a TLS socket can hang with pending data available in the
> > NIC that is not delivered to the host. As a workaround, TLS sockets
> > are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
> > any data is read from a TLS socket. In addition, a fallback timer will
> > periodically send CPL_RX_DATA_ACK messages to the NIC for connections
> > that are still in the handshake phase. Once the connection has
> > finished the handshake and programmed RX keys via the socket option,
> > the timer is stopped.
> >
> > A new function select_ulp_mode() is used to determine what sub-mode a
> > given TOE socket should use (plain TOE, DDP, or TLS). The existing
> > set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
> > handles initialization of TLS-specific state when necessary in
> > addition to DDP-specific state.
> >
> > Since TLS sockets do not receive individual TCP segments but always
> > receive full TLS records, they can receive more data than is available
> > in the current window (e.g. if a 16k TLS record is received but the
> > socket buffer is itself 16k). To cope with this, just drop the window
> > to 0 when this happens, but track the overage and "eat" the overage as
> > it is read from the socket buffer not opening the window (or adding
> > rx_credits) for the overage bytes.
> >
> > Reviewed by: np (earlier version)
> > Sponsored by: Chelsio Communications
> > Differential Revision: https://reviews.freebsd.org/D14529
> >
> > Added:
> > head/sys/dev/cxgbe/tom/t4_tls.c (contents, props changed)
> > head/sys/dev/cxgbe/tom/t4_tls.h (contents, props changed)
> > Modified:
> > head/sys/dev/cxgbe/adapter.h
> > head/sys/dev/cxgbe/firmware/t6fw_cfg.txt
> > head/sys/dev/cxgbe/offload.h
> > head/sys/dev/cxgbe/t4_main.c
> > head/sys/dev/cxgbe/tom/t4_connect.c
> > head/sys/dev/cxgbe/tom/t4_cpl_io.c
> > head/sys/dev/cxgbe/tom/t4_listen.c
> > head/sys/dev/cxgbe/tom/t4_tom.c
> > head/sys/dev/cxgbe/tom/t4_tom.h
> > head/sys/modules/cxgbe/tom/Makefile
> >
> > Modified: head/sys/dev/cxgbe/adapter.h
> > ==============================================================================
> > --- head/sys/dev/cxgbe/adapter.h Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/adapter.h Tue Mar 13 23:05:51
> > 2018 (r330884) @@ -297,6 +297,10 @@ struct port_info {
> > struct port_stats stats;
> > u_int tnl_cong_drops;
> > u_int tx_parse_error;
> > + u_long tx_tls_records;
> > + u_long tx_tls_octets;
> > + u_long rx_tls_records;
> > + u_long rx_tls_octets;
> >
> > struct callout tick;
> > };
> >
> > Modified: head/sys/dev/cxgbe/firmware/t6fw_cfg.txt
> > ==============================================================================
> > --- head/sys/dev/cxgbe/firmware/t6fw_cfg.txt Tue Mar 13 22:54:29
> > 2018 (r330883) +++ head/sys/dev/cxgbe/firmware/t6fw_cfg.txt
> > Tue Mar 13 23:05:51 2018 (r330884) @@ -163,10 +163,12 @@
> > nserver = 512
> > nhpfilter = 0
> > nhash = 16384
> > - protocol = ofld, rddp, rdmac, iscsi_initiator_pdu,
> > iscsi_target_pdu, iscsi_t10dif, crypto_lookaside
> > + protocol = ofld, rddp, rdmac, iscsi_initiator_pdu,
> > iscsi_target_pdu, iscsi_t10dif, tlskeys, crypto_lookaside tp_l2t = 4096
> > tp_ddp = 2
> > tp_ddp_iscsi = 2
> > + tp_tls_key = 3
> > + tp_tls_mxrxsize = 17408 # 16384 + 1024, governs max rx data, pm
> > max xfer len, rx coalesce sizes tp_stag = 2
> > tp_pbl = 5
> > tp_rq = 7
> > @@ -273,7 +275,7 @@
> >
> > [fini]
> > version = 0x1
> > - checksum = 0x7191019f
> > + checksum = 0x9e8952d2
> > #
> > # $FreeBSD$
> > #
> >
> > Modified: head/sys/dev/cxgbe/offload.h
> > ==============================================================================
> > --- head/sys/dev/cxgbe/offload.h Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/offload.h Tue Mar 13 23:05:51
> > 2018 (r330884) @@ -151,6 +151,9 @@ struct tom_tunables {
> > int sndbuf;
> > int ddp;
> > int rx_coalesce;
> > + int tls;
> > + int *tls_rx_ports;
> > + int num_tls_rx_ports;
> > int tx_align;
> > int tx_zcopy;
> > };
> >
> > Modified: head/sys/dev/cxgbe/t4_main.c
> > ==============================================================================
> > --- head/sys/dev/cxgbe/t4_main.c Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/t4_main.c Tue Mar 13 23:05:51
> > 2018 (r330884) @@ -591,6 +591,7 @@ static int
> > sysctl_wcwr_stats(SYSCTL_HANDLER_ARGS); static int
> > sysctl_tc_params(SYSCTL_HANDLER_ARGS); #endif
> > #ifdef TCP_OFFLOAD
> > +static int sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS);
> > static int sysctl_tp_tick(SYSCTL_HANDLER_ARGS);
> > static int sysctl_tp_dack_timer(SYSCTL_HANDLER_ARGS);
> > static int sysctl_tp_timer(SYSCTL_HANDLER_ARGS);
> > @@ -1390,6 +1391,7 @@ t4_detach_common(device_t dev)
> > free(sc->sge.iqmap, M_CXGBE);
> > free(sc->sge.eqmap, M_CXGBE);
> > free(sc->tids.ftid_tab, M_CXGBE);
> > + free(sc->tt.tls_rx_ports, M_CXGBE);
> > t4_destroy_dma_tag(sc);
> > if (mtx_initialized(&sc->sc_lock)) {
> > sx_xlock(&t4_list_lock);
> > @@ -5433,6 +5435,14 @@ t4_sysctls(struct adapter *sc)
> > SYSCTL_ADD_INT(ctx, children, OID_AUTO, "rx_coalesce",
> > CTLFLAG_RW, &sc->tt.rx_coalesce, 0, "receive
> > coalescing");
> > + sc->tt.tls = 0;
> > + SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tls", CTLFLAG_RW,
> > + &sc->tt.tls, 0, "Inline TLS allowed");
> > +
> > + SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "tls_rx_ports",
> > + CTLTYPE_INT | CTLFLAG_RW, sc, 0, sysctl_tls_rx_ports,
> > + "I", "TCP ports that use inline TLS+TOE RX");
> > +
> > sc->tt.tx_align = 1;
> > SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_align",
> > CTLFLAG_RW, &sc->tt.tx_align, 0, "chop and align
> > payload"); @@ -5836,6 +5846,19 @@ cxgbe_sysctls(struct port_info *pi)
> > "# of buffer-group 3 truncated packets");
> >
> > #undef SYSCTL_ADD_T4_PORTSTAT
> > +
> > + SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_records",
> > + CTLFLAG_RD, &pi->tx_tls_records,
> > + "# of TLS records transmitted");
> > + SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_octets",
> > + CTLFLAG_RD, &pi->tx_tls_octets,
> > + "# of payload octets in transmitted TLS records");
> > + SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_records",
> > + CTLFLAG_RD, &pi->rx_tls_records,
> > + "# of TLS records received");
> > + SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_octets",
> > + CTLFLAG_RD, &pi->rx_tls_octets,
> > + "# of payload octets in received TLS records");
> > }
> >
> > static int
> > @@ -8257,6 +8280,68 @@ done:
> > #endif
> >
> > #ifdef TCP_OFFLOAD
> > +static int
> > +sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS)
> > +{
> > + struct adapter *sc = arg1;
> > + int *old_ports, *new_ports;
> > + int i, new_count, rc;
> > +
> > + if (req->newptr == NULL && req->oldptr == NULL)
> > + return (SYSCTL_OUT(req, NULL, imax(sc->tt.num_tls_rx_ports,
> > 1) *
> > + sizeof(sc->tt.tls_rx_ports[0])));
> > +
> > + rc = begin_synchronized_op(sc, NULL, SLEEP_OK | INTR_OK,
> > "t4tlsrx");
> > + if (rc)
> > + return (rc);
> > +
> > + if (sc->tt.num_tls_rx_ports == 0) {
> > + i = -1;
> > + rc = SYSCTL_OUT(req, &i, sizeof(i));
> > + } else
> > + rc = SYSCTL_OUT(req, sc->tt.tls_rx_ports,
> > + sc->tt.num_tls_rx_ports *
> > sizeof(sc->tt.tls_rx_ports[0]));
> > + if (rc == 0 && req->newptr != NULL) {
> > + new_count = req->newlen / sizeof(new_ports[0]);
> > + new_ports = malloc(new_count * sizeof(new_ports[0]),
> > M_CXGBE,
> > + M_WAITOK);
> > + rc = SYSCTL_IN(req, new_ports, new_count *
> > + sizeof(new_ports[0]));
> > + if (rc)
> > + goto err;
> > +
> > + /* Allow setting to a single '-1' to clear the list. */
> > + if (new_count == 1 && new_ports[0] == -1) {
> > + ADAPTER_LOCK(sc);
> > + old_ports = sc->tt.tls_rx_ports;
> > + sc->tt.tls_rx_ports = NULL;
> > + sc->tt.num_tls_rx_ports = 0;
> > + ADAPTER_UNLOCK(sc);
> > + free(old_ports, M_CXGBE);
> > + } else {
> > + for (i = 0; i < new_count; i++) {
> > + if (new_ports[i] < 1 ||
> > + new_ports[i] > IPPORT_MAX) {
> > + rc = EINVAL;
> > + goto err;
> > + }
> > + }
> > +
> > + ADAPTER_LOCK(sc);
> > + old_ports = sc->tt.tls_rx_ports;
> > + sc->tt.tls_rx_ports = new_ports;
> > + sc->tt.num_tls_rx_ports = new_count;
> > + ADAPTER_UNLOCK(sc);
> > + free(old_ports, M_CXGBE);
> > + new_ports = NULL;
> > + }
> > + err:
> > + free(new_ports, M_CXGBE);
> > + }
> > + end_synchronized_op(sc, 0);
> > + return (rc);
> > +}
> > +
> > static void
> > unit_conv(char *buf, size_t len, u_int val, u_int factor)
> > {
> >
> > Modified: head/sys/dev/cxgbe/tom/t4_connect.c
> > ==============================================================================
> > --- head/sys/dev/cxgbe/tom/t4_connect.c Tue Mar 13 22:54:29
> > 2018 (r330883) +++ head/sys/dev/cxgbe/tom/t4_connect.c Tue Mar
> > 13 23:05:51 2018 (r330884) @@ -142,6 +142,10 @@
> > do_act_establish(struct sge_iq *iq, const struct rss_h }
> >
> > make_established(toep, cpl->snd_isn, cpl->rcv_isn, cpl->tcp_opt);
> > +
> > + if (toep->ulp_mode == ULP_MODE_TLS)
> > + tls_establish(toep);
> > +
> > done:
> > INP_WUNLOCK(inp);
> > CURVNET_RESTORE();
> > @@ -268,6 +272,11 @@ calc_opt2a(struct socket *so, struct toepcb *toep)
> > if (toep->ulp_mode == ULP_MODE_TCPDDP)
> > opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
> > #endif
> > + if (toep->ulp_mode == ULP_MODE_TLS) {
> > + opt2 |= F_RX_FC_VALID;
> > + opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
> > + opt2 |= F_RX_FC_DISABLE;
> > + }
> >
> > return (htobe32(opt2));
> > }
> > @@ -378,10 +387,7 @@ t4_connect(struct toedev *tod, struct socket *so, stru
> > DONT_OFFLOAD_ACTIVE_OPEN(ENOMEM);
> >
> > toep->vnet = so->so_vnet;
> > - if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0)
> > - set_tcpddp_ulp_mode(toep);
> > - else
> > - toep->ulp_mode = ULP_MODE_NONE;
> > + set_ulp_mode(toep, select_ulp_mode(so, sc));
> > SOCKBUF_LOCK(&so->so_rcv);
> > /* opt0 rcv_bufsiz initially, assumes its normal meaning later */
> > toep->rx_credits = min(select_rcv_wnd(so) >> 10, M_RCV_BUFSIZ);
> >
> > Modified: head/sys/dev/cxgbe/tom/t4_cpl_io.c
> > ==============================================================================
> > --- head/sys/dev/cxgbe/tom/t4_cpl_io.c Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/tom/t4_cpl_io.c Tue Mar 13 23:05:51
> > 2018 (r330884) @@ -73,9 +73,6 @@ __FBSDID("$FreeBSD$");
> > #include "tom/t4_tom_l2t.h"
> > #include "tom/t4_tom.h"
> >
> > -#define
> > IS_AIOTX_MBUF(m) \
> > - ((m)->m_flags & M_EXT && (m)->m_ext.ext_flags & EXT_FLAG_AIOTX)
> > -
> > static void t4_aiotx_cancel(struct kaiocb *job);
> > static void t4_aiotx_queue_toep(struct toepcb *toep);
> >
> > @@ -106,7 +103,7 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
> > {
> > struct wrqe *wr;
> > struct fw_flowc_wr *flowc;
> > - unsigned int nparams = ftxp ? 8 : 6, flowclen;
> > + unsigned int nparams, flowclen, paramidx;
> > struct vi_info *vi = toep->vi;
> > struct port_info *pi = vi->pi;
> > struct adapter *sc = pi->adapter;
> > @@ -116,6 +113,15 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
> > KASSERT(!(toep->flags & TPF_FLOWC_WR_SENT),
> > ("%s: flowc for tid %u sent already", __func__, toep->tid));
> >
> > + if (ftxp != NULL)
> > + nparams = 8;
> > + else
> > + nparams = 6;
> > + if (toep->ulp_mode == ULP_MODE_TLS)
> > + nparams++;
> > + if (toep->tls.fcplenmax != 0)
> > + nparams++;
> > +
> > flowclen = sizeof(*flowc) + nparams * sizeof(struct
> > fw_flowc_mnemval);
> > wr = alloc_wrqe(roundup2(flowclen, 16), toep->ofld_txq);
> > @@ -131,39 +137,45 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
> > flowc->flowid_len16 = htonl(V_FW_WR_LEN16(howmany(flowclen, 16)) |
> > V_FW_WR_FLOWID(toep->tid));
> >
> > - flowc->mnemval[0].mnemonic = FW_FLOWC_MNEM_PFNVFN;
> > - flowc->mnemval[0].val = htobe32(pfvf);
> > - flowc->mnemval[1].mnemonic = FW_FLOWC_MNEM_CH;
> > - flowc->mnemval[1].val = htobe32(pi->tx_chan);
> > - flowc->mnemval[2].mnemonic = FW_FLOWC_MNEM_PORT;
> > - flowc->mnemval[2].val = htobe32(pi->tx_chan);
> > - flowc->mnemval[3].mnemonic = FW_FLOWC_MNEM_IQID;
> > - flowc->mnemval[3].val = htobe32(toep->ofld_rxq->iq.abs_id);
> > +#define FLOWC_PARAM(__m, __v) \
> > + do { \
> > + flowc->mnemval[paramidx].mnemonic = FW_FLOWC_MNEM_##__m; \
> > + flowc->mnemval[paramidx].val = htobe32(__v); \
> > + paramidx++; \
> > + } while (0)
> > +
> > + paramidx = 0;
> > +
> > + FLOWC_PARAM(PFNVFN, pfvf);
> > + FLOWC_PARAM(CH, pi->tx_chan);
> > + FLOWC_PARAM(PORT, pi->tx_chan);
> > + FLOWC_PARAM(IQID, toep->ofld_rxq->iq.abs_id);
> > if (ftxp) {
> > uint32_t sndbuf = min(ftxp->snd_space, sc->tt.sndbuf);
> >
> > - flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDNXT;
> > - flowc->mnemval[4].val = htobe32(ftxp->snd_nxt);
> > - flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_RCVNXT;
> > - flowc->mnemval[5].val = htobe32(ftxp->rcv_nxt);
> > - flowc->mnemval[6].mnemonic = FW_FLOWC_MNEM_SNDBUF;
> > - flowc->mnemval[6].val = htobe32(sndbuf);
> > - flowc->mnemval[7].mnemonic = FW_FLOWC_MNEM_MSS;
> > - flowc->mnemval[7].val = htobe32(ftxp->mss);
> > + FLOWC_PARAM(SNDNXT, ftxp->snd_nxt);
> > + FLOWC_PARAM(RCVNXT, ftxp->rcv_nxt);
> > + FLOWC_PARAM(SNDBUF, sndbuf);
> > + FLOWC_PARAM(MSS, ftxp->mss);
> >
> > CTR6(KTR_CXGBE,
> > "%s: tid %u, mss %u, sndbuf %u, snd_nxt 0x%x, rcv_nxt
> > 0x%x", __func__, toep->tid, ftxp->mss, sndbuf, ftxp->snd_nxt,
> > ftxp->rcv_nxt);
> > } else {
> > - flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDBUF;
> > - flowc->mnemval[4].val = htobe32(512);
> > - flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_MSS;
> > - flowc->mnemval[5].val = htobe32(512);
> > + FLOWC_PARAM(SNDBUF, 512);
> > + FLOWC_PARAM(MSS, 512);
> >
> > CTR2(KTR_CXGBE, "%s: tid %u", __func__, toep->tid);
> > }
> > + if (toep->ulp_mode == ULP_MODE_TLS)
> > + FLOWC_PARAM(ULP_MODE, toep->ulp_mode);
> > + if (toep->tls.fcplenmax != 0)
> > + FLOWC_PARAM(TXDATAPLEN_MAX, toep->tls.fcplenmax);
> > +#undef FLOWC_PARAM
> >
> > + KASSERT(paramidx == nparams, ("nparams mismatch"));
> > +
> > txsd->tx_credits = howmany(flowclen, 16);
> > txsd->plen = 0;
> > KASSERT(toep->tx_credits >= txsd->tx_credits && toep->txsd_avail >
> > 0, @@ -421,7 +433,7 @@ make_established(struct toepcb *toep, uint32_t
> > snd_isn soisconnected(so);
> > }
> >
> > -static int
> > +int
> > send_rx_credits(struct adapter *sc, struct toepcb *toep, int credits)
> > {
> > struct wrqe *wr;
> > @@ -443,6 +455,23 @@ send_rx_credits(struct adapter *sc, struct toepcb *toe
> > }
> >
> > void
> > +send_rx_modulate(struct adapter *sc, struct toepcb *toep)
> > +{
> > + struct wrqe *wr;
> > + struct cpl_rx_data_ack *req;
> > +
> > + wr = alloc_wrqe(sizeof(*req), toep->ctrlq);
> > + if (wr == NULL)
> > + return;
> > + req = wrtod(wr);
> > +
> > + INIT_TP_WR_MIT_CPL(req, CPL_RX_DATA_ACK, toep->tid);
> > + req->credit_dack = htobe32(F_RX_MODULATE_RX);
> > +
> > + t4_wrq_tx(sc, wr);
> > +}
> > +
> > +void
> > t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
> > {
> > struct adapter *sc = tod->tod_softc;
> > @@ -459,8 +488,18 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
> > ("%s: sb %p has more data (%d) than last time (%d).",
> > __func__, sb, sbused(sb), toep->sb_cc));
> >
> > - toep->rx_credits += toep->sb_cc - sbused(sb);
> > + credits = toep->sb_cc - sbused(sb);
> > toep->sb_cc = sbused(sb);
> > + if (toep->ulp_mode == ULP_MODE_TLS) {
> > + if (toep->tls.rcv_over >= credits) {
> > + toep->tls.rcv_over -= credits;
> > + credits = 0;
> > + } else {
> > + credits -= toep->tls.rcv_over;
> > + toep->tls.rcv_over = 0;
> > + }
> > + }
> > + toep->rx_credits += credits;
> >
> > if (toep->rx_credits > 0 &&
> > (tp->rcv_wnd <= 32 * 1024 || toep->rx_credits >= 64 * 1024 ||
> > @@ -471,7 +510,8 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
> > toep->rx_credits -= credits;
> > tp->rcv_wnd += credits;
> > tp->rcv_adv += credits;
> > - }
> > + } else if (toep->flags & TPF_FORCE_CREDITS)
> > + send_rx_modulate(sc, toep);
> > }
> >
> > void
> > @@ -489,8 +529,8 @@ t4_rcvd(struct toedev *tod, struct tcpcb *tp)
> > /*
> > * Close a connection by sending a CPL_CLOSE_CON_REQ message.
> > */
> > -static int
> > -close_conn(struct adapter *sc, struct toepcb *toep)
> > +int
> > +t4_close_conn(struct adapter *sc, struct toepcb *toep)
> > {
> > struct wrqe *wr;
> > struct cpl_close_con_req *req;
> > @@ -691,6 +731,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep
> >
> > KASSERT(toep->ulp_mode == ULP_MODE_NONE ||
> > toep->ulp_mode == ULP_MODE_TCPDDP ||
> > + toep->ulp_mode == ULP_MODE_TLS ||
> > toep->ulp_mode == ULP_MODE_RDMA,
> > ("%s: ulp_mode %u for toep %p", __func__, toep->ulp_mode,
> > toep));
> > @@ -905,7 +946,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep
> >
> > /* Send a FIN if requested, but only if there's no more data to
> > send */ if (m == NULL && toep->flags & TPF_SEND_FIN)
> > - close_conn(sc, toep);
> > + t4_close_conn(sc, toep);
> > }
> >
> > static inline void
> > @@ -1097,7 +1138,7 @@ t4_push_pdus(struct adapter *sc, struct toepcb *toep,
> >
> > /* Send a FIN if requested, but only if there are no more PDUs to
> > send */ if (mbufq_first(pduq) == NULL && toep->flags & TPF_SEND_FIN)
> > - close_conn(sc, toep);
> > + t4_close_conn(sc, toep);
> > }
> >
> > int
> > @@ -1116,6 +1157,8 @@ t4_tod_output(struct toedev *tod, struct tcpcb *tp)
> >
> > if (toep->ulp_mode == ULP_MODE_ISCSI)
> > t4_push_pdus(sc, toep, 0);
> > + else if (tls_tx_key(toep))
> > + t4_push_tls_records(sc, toep, 0);
> > else
> > t4_push_frames(sc, toep, 0);
> >
> > @@ -1140,6 +1183,8 @@ t4_send_fin(struct toedev *tod, struct tcpcb *tp)
> > if (tp->t_state >= TCPS_ESTABLISHED) {
> > if (toep->ulp_mode == ULP_MODE_ISCSI)
> > t4_push_pdus(sc, toep, 0);
> > + else if (tls_tx_key(toep))
> > + t4_push_tls_records(sc, toep, 0);
> > else
> > t4_push_frames(sc, toep, 0);
> > }
> > @@ -1772,6 +1817,10 @@ do_fw4_ack(struct sge_iq *iq, const struct
> > rss_header credits -= txsd->tx_credits;
> > toep->tx_credits += txsd->tx_credits;
> > plen += txsd->plen;
> > + if (txsd->iv_buffer) {
> > + free(txsd->iv_buffer, M_CXGBE);
> > + txsd->iv_buffer = NULL;
> > + }
> > txsd++;
> > toep->txsd_avail++;
> > KASSERT(toep->txsd_avail <= toep->txsd_total,
> > @@ -1797,6 +1846,8 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header
> > CURVNET_SET(toep->vnet);
> > if (toep->ulp_mode == ULP_MODE_ISCSI)
> > t4_push_pdus(sc, toep, plen);
> > + else if (tls_tx_key(toep))
> > + t4_push_tls_records(sc, toep, plen);
> > else
> > t4_push_frames(sc, toep, plen);
> > CURVNET_RESTORE();
> > @@ -1826,6 +1877,12 @@ do_fw4_ack(struct sge_iq *iq, const struct
> > rss_header tid, plen);
> > #endif
> > sbdrop_locked(sb, plen);
> > + if (tls_tx_key(toep)) {
> > + struct tls_ofld_info *tls_ofld =
> > &toep->tls; +
> > + MPASS(tls_ofld->sb_off >= plen);
> > + tls_ofld->sb_off -= plen;
> > + }
> > if (!TAILQ_EMPTY(&toep->aiotx_jobq))
> > t4_aiotx_queue_toep(toep);
> > sowwakeup_locked(so); /* unlocks so_snd */
> > @@ -2298,6 +2355,9 @@ t4_aio_queue_aiotx(struct socket *so, struct kaiocb *j
> > return (EOPNOTSUPP);
> >
> > if (!sc->tt.tx_zcopy)
> > + return (EOPNOTSUPP);
> > +
> > + if (is_tls_offload(toep) || tls_tx_key(toep))
> > return (EOPNOTSUPP);
> >
> > SOCKBUF_LOCK(&so->so_snd);
> >
> > Modified: head/sys/dev/cxgbe/tom/t4_listen.c
> > ==============================================================================
> > --- head/sys/dev/cxgbe/tom/t4_listen.c Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/tom/t4_listen.c Tue Mar 13 23:05:51
> > 2018 (r330884) @@ -1056,6 +1056,11 @@ calc_opt2p(struct adapter *sc,
> > struct port_info *pi, i if (ulp_mode == ULP_MODE_TCPDDP)
> > opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
> > #endif
> > + if (ulp_mode == ULP_MODE_TLS) {
> > + opt2 |= F_RX_FC_VALID;
> > + opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
> > + opt2 |= F_RX_FC_DISABLE;
> > + }
> >
> > return htobe32(opt2);
> > }
> > @@ -1347,11 +1352,15 @@ found:
> >
> > INIT_TP_WR_MIT_CPL(rpl5, CPL_PASS_ACCEPT_RPL, tid);
> > }
> > - if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0) {
> > - ulp_mode = ULP_MODE_TCPDDP;
> > + ulp_mode = select_ulp_mode(so, sc);
> > + switch (ulp_mode) {
> > + case ULP_MODE_TCPDDP:
> > synqe->flags |= TPF_SYNQE_TCPDDP;
> > - } else
> > - ulp_mode = ULP_MODE_NONE;
> > + break;
> > + case ULP_MODE_TLS:
> > + synqe->flags |= TPF_SYNQE_TLS;
> > + break;
> > + }
> > rpl->opt0 = calc_opt0(so, vi, e, mtu_idx, rscale, rx_credits,
> > ulp_mode); rpl->opt2 = calc_opt2p(sc, pi, rxqid, &cpl->tcpopt, &th,
> > ulp_mode);
> > @@ -1407,8 +1416,8 @@ found:
> > REJECT_PASS_ACCEPT();
> > }
> >
> > - CTR5(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK",
> > - __func__, stid, tid, lctx, synqe);
> > + CTR6(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK
> > mode %d",
> > + __func__, stid, tid, lctx, synqe, ulp_mode);
> >
> > INP_WLOCK(inp);
> > synqe->flags |= TPF_SYNQE_HAS_L2TE;
> > @@ -1557,9 +1566,11 @@ reset:
> > toep->tid = tid;
> > toep->l2te = &sc->l2t->l2tab[synqe->l2e_idx];
> > if (synqe->flags & TPF_SYNQE_TCPDDP)
> > - set_tcpddp_ulp_mode(toep);
> > + set_ulp_mode(toep, ULP_MODE_TCPDDP);
> > + else if (synqe->flags & TPF_SYNQE_TLS)
> > + set_ulp_mode(toep, ULP_MODE_TLS);
> > else
> > - toep->ulp_mode = ULP_MODE_NONE;
> > + set_ulp_mode(toep, ULP_MODE_NONE);
> > /* opt0 rcv_bufsiz initially, assumes its normal meaning later */
> > toep->rx_credits = synqe->rcv_bufsize;
> >
> >
> > Added: head/sys/dev/cxgbe/tom/t4_tls.c
> > ==============================================================================
> > --- /dev/null 00:00:00 1970 (empty, because file is newly
> > added) +++ head/sys/dev/cxgbe/tom/t4_tls.c Tue Mar 13 23:05:51 2018
> > (r330884) @@ -0,0 +1,1642 @@
> > +/*-
> > + * SPDX-License-Identifier: BSD-2-Clause-FreeBSD
> > + *
> > + * Copyright (c) 2017-2018 Chelsio Communications, Inc.
> > + * All rights reserved.
> > + * Written by: John Baldwin <jhb at FreeBSD.org>
> > + *
> > + * Redistribution and use in source and binary forms, with or without
> > + * modification, are permitted provided that the following conditions
> > + * are met:
> > + * 1. Redistributions of source code must retain the above copyright
> > + * notice, this list of conditions and the following disclaimer.
> > + * 2. Redistributions in binary form must reproduce the above copyright
> > + * notice, this list of conditions and the following disclaimer in the
> > + * documentation and/or other materials provided with the distribution.
> > + *
> > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
> > PURPOSE
> > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
> > CONSEQUENTIAL
> > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
> > STRICT
> > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY
> > WAY
> > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> > + * SUCH DAMAGE.
> > + */
> > +
> > +#include "opt_inet.h"
> > +
> > +#include <sys/cdefs.h>
> > +__FBSDID("$FreeBSD$");
> > +
> > +#include <sys/param.h>
> > +#include <sys/sglist.h>
> > +#include <sys/socket.h>
> > +#include <sys/socketvar.h>
> > +#include <sys/systm.h>
> > +#include <netinet/in.h>
> > +#include <netinet/in_pcb.h>
> > +#include <netinet/tcp_var.h>
> > +#include <netinet/toecore.h>
> > +
> > +#ifdef TCP_OFFLOAD
> > +#include "common/common.h"
> > +#include "common/t4_tcb.h"
> > +#include "tom/t4_tom_l2t.h"
> > +#include "tom/t4_tom.h"
> > +
> > +/*
> > + * The TCP sequence number of a CPL_TLS_DATA mbuf is saved here while
> > + * the mbuf is in the ulp_pdu_reclaimq.
> > + */
> > +#define tls_tcp_seq PH_loc.thirtytwo[0]
> > +
> > +/*
> > + * Handshake lock used for the handshake timer. Having a global lock
> > + * is perhaps not ideal, but it avoids having to use callout_drain()
> > + * in tls_uninit_toep() which can't block. Also, the timer shouldn't
> > + * actually fire for most connections.
> > + */
> > +static struct mtx tls_handshake_lock;
> > +
> > +static void
> > +t4_set_tls_tcb_field(struct toepcb *toep, uint16_t word, uint64_t mask,
> > + uint64_t val)
> > +{
> > + struct adapter *sc = td_adapter(toep->td);
> > +
> > + t4_set_tcb_field(sc, toep->ctrlq, toep->tid, word, mask, val, 0, 0,
> > + toep->ofld_rxq->iq.abs_id);
> > +}
> > +
> > +/* TLS and DTLS common routines */
> > +int
> > +tls_tx_key(struct toepcb *toep)
> > +{
> > + struct tls_ofld_info *tls_ofld = &toep->tls;
> > +
> > + return (tls_ofld->tx_key_addr >= 0);
> > +}
> > +
> > +int
> > +tls_rx_key(struct toepcb *toep)
> > +{
> > + struct tls_ofld_info *tls_ofld = &toep->tls;
> > +
> > + return (tls_ofld->rx_key_addr >= 0);
> > +}
> > +
> > +static int
> > +key_size(struct toepcb *toep)
> > +{
> > + struct tls_ofld_info *tls_ofld = &toep->tls;
> > +
> > + return ((tls_ofld->key_location ==
> > TLS_SFO_WR_CONTEXTLOC_IMMEDIATE) ?
> > + tls_ofld->k_ctx.tx_key_info_size : KEY_IN_DDR_SIZE);
> > +}
> > +
> > +/* Set TLS Key-Id in TCB */
> > +static void
> > +t4_set_tls_keyid(struct toepcb *toep, unsigned int key_id)
> > +{
> > +
> > + t4_set_tls_tcb_field(toep, W_TCB_RX_TLS_KEY_TAG,
> > + V_TCB_RX_TLS_KEY_TAG(M_TCB_RX_TLS_BUF_TAG),
> > + V_TCB_RX_TLS_KEY_TAG(key_id));
> > +}
> > +
> > +/* Clear TF_RX_QUIESCE to re-enable receive. */
> > +static void
> > +t4_clear_rx_quiesce(struct toepcb *toep)
> > +{
> > +
> > + t4_set_tls_tcb_field(toep, W_TCB_T_FLAGS, V_TF_RX_QUIESCE(1), 0);
> > +}
> > +
> > +static void
> > +tls_clr_ofld_mode(struct toepcb *toep)
> > +{
> > +
> > + tls_stop_handshake_timer(toep);
> > +
> > + /* Operate in PDU extraction mode only. */
> > + t4_set_tls_tcb_field(toep, W_TCB_ULP_RAW,
> > + V_TCB_ULP_RAW(M_TCB_ULP_RAW),
> > + V_TCB_ULP_RAW(V_TF_TLS_ENABLE(1)));
> > + t4_clear_rx_quiesce(toep);
> > +}
> > +
> > +static void
> > +tls_clr_quiesce(struct toepcb *toep)
> > +{
> > +
> > + tls_stop_handshake_timer(toep);
> > + t4_clear_rx_quiesce(toep);
> > +}
> > +
> > +/*
> > + * Calculate the TLS data expansion size
> > + */
> > +static int
> > +tls_expansion_size(struct toepcb *toep, int data_len, int full_pdus_only,
> > + unsigned short *pdus_per_ulp)
> > +{
> > + struct tls_ofld_info *tls_ofld = &toep->tls;
> > + struct tls_scmd *scmd = &tls_ofld->scmd0;
> > + int expn_size = 0, frag_count = 0, pad_per_pdu = 0,
> > + pad_last_pdu = 0, last_frag_size = 0, max_frag_size = 0;
> > + int exp_per_pdu = 0;
> > + int hdr_len = TLS_HEADER_LENGTH;
> > +
> > + do {
> > + max_frag_size = tls_ofld->k_ctx.frag_size;
> > + if (G_SCMD_CIPH_MODE(scmd->seqno_numivs) ==
> > + SCMD_CIPH_MODE_AES_GCM) {
> > + frag_count = (data_len / max_frag_size);
> > + exp_per_pdu = GCM_TAG_SIZE +
> > AEAD_EXPLICIT_DATA_SIZE +
> > + hdr_len;
> > + expn_size = frag_count * exp_per_pdu;
> > + if (full_pdus_only) {
> > + *pdus_per_ulp = data_len / (exp_per_pdu +
> > + max_frag_size);
> > + if (*pdus_per_ulp > 32)
> > + *pdus_per_ulp = 32;
> > + else if(!*pdus_per_ulp)
> > + *pdus_per_ulp = 1;
> > + expn_size = (*pdus_per_ulp) * exp_per_pdu;
> > + break;
> > + }
> > + if ((last_frag_size = data_len % max_frag_size) >
> > 0) {
> > + frag_count += 1;
> > + expn_size += exp_per_pdu;
> > + }
> > + break;
> > + } else if (G_SCMD_CIPH_MODE(scmd->seqno_numivs) !=
> > + SCMD_CIPH_MODE_NOP) {
> > + /* Calculate the number of fragments we can make */
> > + frag_count = (data_len / max_frag_size);
> > + if (frag_count > 0) {
> > + pad_per_pdu = (((howmany((max_frag_size +
> > +
> > tls_ofld->mac_length),
> > + CIPHER_BLOCK_SIZE)) *
> > + CIPHER_BLOCK_SIZE) -
> > + (max_frag_size +
> > + tls_ofld->mac_length));
> > + if (!pad_per_pdu)
> > + pad_per_pdu = CIPHER_BLOCK_SIZE;
> > + exp_per_pdu = pad_per_pdu +
> > + tls_ofld->mac_length +
> > + hdr_len + CIPHER_BLOCK_SIZE;
> > + expn_size = frag_count * exp_per_pdu;
> > + }
> > + if (full_pdus_only) {
> > + *pdus_per_ulp = data_len / (exp_per_pdu +
> > + max_frag_size);
> > + if (*pdus_per_ulp > 32)
> > + *pdus_per_ulp = 32;
> > + else if (!*pdus_per_ulp)
> > + *pdus_per_ulp = 1;
> > + expn_size = (*pdus_per_ulp) * exp_per_pdu;
> > + break;
> > + }
> > + /* Consider the last fragment */
> > + if ((last_frag_size = data_len % max_frag_size) >
> > 0) {
> > + pad_last_pdu = (((howmany((last_frag_size +
> > +
> > tls_ofld->mac_length),
> > + CIPHER_BLOCK_SIZE))
> > *
> > + CIPHER_BLOCK_SIZE) -
> > + (last_frag_size +
> > + tls_ofld->mac_length));
> > + if (!pad_last_pdu)
> > + pad_last_pdu = CIPHER_BLOCK_SIZE;
> > + expn_size += (pad_last_pdu +
> > + tls_ofld->mac_length +
> > hdr_len +
> > + CIPHER_BLOCK_SIZE);
> > + }
> > + }
> > + } while (0);
> > +
> > + return (expn_size);
> > +}
> > +
> > +/* Copy Key to WR */
> > +static void
> > +tls_copy_tx_key(struct toepcb *toep, void *dst)
> > +{
> > + struct tls_ofld_info *tls_ofld = &toep->tls;
> > + struct ulptx_sc_memrd *sc_memrd;
> > + struct ulptx_idata *sc;
> > +
> > + if (tls_ofld->k_ctx.tx_key_info_size <= 0)
> > + return;
> > +
> > + if (tls_ofld->key_location == TLS_SFO_WR_CONTEXTLOC_DDR) {
> > + sc = dst;
> > + sc->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_NOOP));
> > + sc->len = htobe32(0);
> > + sc_memrd = (struct ulptx_sc_memrd *)(sc + 1);
> > + sc_memrd->cmd_to_len =
> > htobe32(V_ULPTX_CMD(ULP_TX_SC_MEMRD) |
> > + V_ULP_TX_SC_MORE(1) |
> > + V_ULPTX_LEN16(tls_ofld->k_ctx.tx_key_info_size >> 4));
> > + sc_memrd->addr = htobe32(tls_ofld->tx_key_addr >> 5);
> > + } else if (tls_ofld->key_location ==
> > TLS_SFO_WR_CONTEXTLOC_IMMEDIATE) {
> > + memcpy(dst, &tls_ofld->k_ctx.tx,
> > + tls_ofld->k_ctx.tx_key_info_size);
> > + }
> > +}
> > +
> > +/* TLS/DTLS content type for CPL SFO */
> > +static inline unsigned char
> > +tls_content_type(unsigned char content_type)
> > +{
> > + /*
> > + * XXX: Shouldn't this map CONTENT_TYPE_APP_DATA to DATA and
> > + * default to "CUSTOM" for all other types including
> > + * heartbeat?
> > + */
> > + switch (content_type) {
> > + case CONTENT_TYPE_CCS:
> > + return CPL_TX_TLS_SFO_TYPE_CCS;
> > + case CONTENT_TYPE_ALERT:
> > + return CPL_TX_TLS_SFO_TYPE_ALERT;
> > + case CONTENT_TYPE_HANDSHAKE:
> > + return CPL_TX_TLS_SFO_TYPE_HANDSHAKE;
> > + case CONTENT_TYPE_HEARTBEAT:
> > + return CPL_TX_TLS_SFO_TYPE_HEARTBEAT;
> > + }
> > + return CPL_TX_TLS_SFO_TYPE_DATA;
> > +}
> > +
> > +static unsigned char
> > +get_cipher_key_size(unsigned int ck_size)
> > +{
> > + switch (ck_size) {
> > + case AES_NOP: /* NOP */
> > + return 15;
> > + case AES_128: /* AES128 */
> > + return CH_CK_SIZE_128;
> > + case AES_192: /* AES192 */
> > + return CH_CK_SIZE_192;
> > + case AES_256: /* AES256 */
> > + return CH_CK_SIZE_256;
> > + default:
> > + return CH_CK_SIZE_256;
> > + }
> > +}
> > +
> > +static unsigned char
> > +get_mac_key_size(unsigned int mk_size)
> > +{
> > + switch (mk_size) {
> > + case SHA_NOP: /* NOP */
> > + return CH_MK_SIZE_128;
> > + case SHA_GHASH: /* GHASH */
> > + case SHA_512: /* SHA512 */
> > + return CH_MK_SIZE_512;
> > + case SHA_224: /* SHA2-224 */
> > + return CH_MK_SIZE_192;
> > + case SHA_256: /* SHA2-256*/
> > + return CH_MK_SIZE_256;
> > + case SHA_384: /* SHA384 */
> > + return CH_MK_SIZE_512;
> > + case SHA1: /* SHA1 */
> > + default:
> > + return CH_MK_SIZE_160;
> > + }
> > +}
> > +
> > +static unsigned int
> > +get_proto_ver(int proto_ver)
> > +{
> > + switch (proto_ver) {
> > + case TLS1_2_VERSION:
> > + return TLS_1_2_VERSION;
> > + case TLS1_1_VERSION:
> > + return TLS_1_1_VERSION;
> > + case DTLS1_2_VERSION:
> > + return DTLS_1_2_VERSION;
> > + default:
> > + return TLS_VERSION_MAX;
> > + }
> > +}
> > +
> > +static void
> > +tls_rxkey_flit1(struct tls_keyctx *kwr, struct tls_key_context *kctx)
> > +{
> > +
> > + if (kctx->state.enc_mode == CH_EVP_CIPH_GCM_MODE) {
> > + kwr->u.rxhdr.ivinsert_to_authinsrt =
> > + htobe64(V_TLS_KEYCTX_TX_WR_IVINSERT(6ULL) |
> > + V_TLS_KEYCTX_TX_WR_AADSTRTOFST(1ULL) |
> > + V_TLS_KEYCTX_TX_WR_AADSTOPOFST(5ULL) |
> > + V_TLS_KEYCTX_TX_WR_AUTHSRTOFST(14ULL) |
> > + V_TLS_KEYCTX_TX_WR_AUTHSTOPOFST(16ULL) |
> > + V_TLS_KEYCTX_TX_WR_CIPHERSRTOFST(14ULL) |
> > + V_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST(0ULL) |
> > + V_TLS_KEYCTX_TX_WR_AUTHINSRT(16ULL));
> > + kwr->u.rxhdr.ivpresent_to_rxmk_size &=
> > + ~(V_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(1));
> > + kwr->u.rxhdr.authmode_to_rxvalid &=
> > + ~(V_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(1));
> > + } else {
> > + kwr->u.rxhdr.ivinsert_to_authinsrt =
> > + htobe64(V_TLS_KEYCTX_TX_WR_IVINSERT(6ULL) |
> > + V_TLS_KEYCTX_TX_WR_AADSTRTOFST(1ULL) |
> > + V_TLS_KEYCTX_TX_WR_AADSTOPOFST(5ULL) |
> > + V_TLS_KEYCTX_TX_WR_AUTHSRTOFST(22ULL) |
> > + V_TLS_KEYCTX_TX_WR_AUTHSTOPOFST(0ULL) |
> > + V_TLS_KEYCTX_TX_WR_CIPHERSRTOFST(22ULL) |
> > + V_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST(0ULL) |
> > + V_TLS_KEYCTX_TX_WR_AUTHINSRT(0ULL));
> > + }
> > +}
> > +
> > +/* Rx key */
> > +static void
> > +prepare_rxkey_wr(struct tls_keyctx *kwr, struct tls_key_context *kctx)
> > +{
> > + unsigned int ck_size = kctx->cipher_secret_size;
> > + unsigned int mk_size = kctx->mac_secret_size;
> > + int proto_ver = kctx->proto_ver;
> > +
> > + kwr->u.rxhdr.flitcnt_hmacctrl =
> > + ((kctx->tx_key_info_size >> 4) << 3) | kctx->hmac_ctrl;
> > +
> > + kwr->u.rxhdr.protover_ciphmode =
> > + V_TLS_KEYCTX_TX_WR_PROTOVER(get_proto_ver(proto_ver)) |
> > + V_TLS_KEYCTX_TX_WR_CIPHMODE(kctx->state.enc_mode);
> > +
> > + kwr->u.rxhdr.authmode_to_rxvalid =
> > + V_TLS_KEYCTX_TX_WR_AUTHMODE(kctx->state.auth_mode) |
> > + V_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(1) |
> > + V_TLS_KEYCTX_TX_WR_SEQNUMCTRL(3) |
> > + V_TLS_KEYCTX_TX_WR_RXVALID(1);
> > +
> > + kwr->u.rxhdr.ivpresent_to_rxmk_size =
> > + V_TLS_KEYCTX_TX_WR_IVPRESENT(0) |
> > + V_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(1) |
> > + V_TLS_KEYCTX_TX_WR_RXCK_SIZE(get_cipher_key_size(ck_size))
> > |
> > + V_TLS_KEYCTX_TX_WR_RXMK_SIZE(get_mac_key_size(mk_size));
> > +
> > + tls_rxkey_flit1(kwr, kctx);
> > +
> > + /* No key reversal for GCM */
> > + if (kctx->state.enc_mode != CH_EVP_CIPH_GCM_MODE) {
> > + t4_aes_getdeckey(kwr->keys.edkey, kctx->rx.key,
> > + (kctx->cipher_secret_size << 3));
> > + memcpy(kwr->keys.edkey + kctx->cipher_secret_size,
> > + kctx->rx.key + kctx->cipher_secret_size,
> > + (IPAD_SIZE + OPAD_SIZE));
> > + } else {
> > + memcpy(kwr->keys.edkey, kctx->rx.key,
> > + (kctx->tx_key_info_size - SALT_SIZE));
> > + memcpy(kwr->u.rxhdr.rxsalt, kctx->rx.salt, SALT_SIZE);
> > + }
> > +}
> > +
> > +/* Tx key */
> > +static void
> > +prepare_txkey_wr(struct tls_keyctx *kwr, struct tls_key_context *kctx)
> > +{
> > + unsigned int ck_size = kctx->cipher_secret_size;
> > + unsigned int mk_size = kctx->mac_secret_size;
> > +
> > + kwr->u.txhdr.ctxlen =
> > + (kctx->tx_key_info_size >> 4);
> > + kwr->u.txhdr.dualck_to_txvalid =
> > + V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1) |
> > + V_TLS_KEYCTX_TX_WR_SALT_PRESENT(1) |
> > + V_TLS_KEYCTX_TX_WR_TXCK_SIZE(get_cipher_key_size(ck_size))
> > |
> > + V_TLS_KEYCTX_TX_WR_TXMK_SIZE(get_mac_key_size(mk_size)) |
> > + V_TLS_KEYCTX_TX_WR_TXVALID(1);
> > +
> > + memcpy(kwr->keys.edkey, kctx->tx.key, HDR_KCTX_SIZE);
> > + if (kctx->state.enc_mode == CH_EVP_CIPH_GCM_MODE) {
> > + memcpy(kwr->u.txhdr.txsalt, kctx->tx.salt, SALT_SIZE);
> > + kwr->u.txhdr.dualck_to_txvalid &=
> > + ~(V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1));
> > + }
> > + kwr->u.txhdr.dualck_to_txvalid =
> > htons(kwr->u.txhdr.dualck_to_txvalid); +}
> > +
> > +/* TLS Key memory management */
> > +int
> > +tls_init_kmap(struct adapter *sc, struct tom_data *td)
> > +{
> > +
> > + td->key_map = vmem_create("T4TLS key map", sc->vres.key.start,
> > + sc->vres.key.size, 8, 0, M_FIRSTFIT | M_NOWAIT);
> > + if (td->key_map == NULL)
> > + return (ENOMEM);
> > + return (0);
> > +}
> > +
> > +void
> > +tls_free_kmap(struct tom_data *td)
> > +{
> > +
> > + if (td->key_map != NULL)
> > + vmem_destroy(td->key_map);
> > +}
> > +
> > +static int
> > +get_new_keyid(struct toepcb *toep, struct tls_key_context *k_ctx)
> > +{
> > + struct tom_data *td = toep->td;
> > + vmem_addr_t addr;
> > +
> > + if (vmem_alloc(td->key_map, TLS_KEY_CONTEXT_SZ, M_NOWAIT |
> > M_FIRSTFIT,
> > + &addr) != 0)
> > + return (-1);
> > +
> > + return (addr);
> > +}
> > +
> > +static void
> > +free_keyid(struct toepcb *toep, int keyid)
> > +{
> > + struct tom_data *td = toep->td;
> > +
> > + vmem_free(td->key_map, keyid, TLS_KEY_CONTEXT_SZ);
> > +}
> > +
> >
> > *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
> > _______________________________________________
> > svn-src-head at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/svn-src-head
> > To unsubscribe, send any mail to "svn-src-head-unsubscribe at freebsd.org"
>
> Building a kernel fails on CURRENT revision 330908 with the follwoing error:
>
> [...]
> ===> gpio/gpioiic (all)
> --- all_subdir_cxgbe ---
> --- cm.o ---
> In file included from /usr/src/sys/dev/cxgbe/iw_cxgbe/cm.c:68:
> /usr/src/sys/dev/cxgbe/tom/t4_tom.h:327:28: error: use of undeclared
> identifier 'ULP_MODE_TLS' return (toep->ulp_mode == ULP_MODE_TLS);
> ^
> /usr/src/sys/dev/cxgbe/tom/t4_tom.h:333:12: error: incomplete definition of
> type 'struct adapter' return (sc->tt.tls && sc->cryptocaps &
> FW_CAPS_CONFIG_TLSKEYS); ~~^
> /usr/src/sys/dev/cxgbe/offload.h:139:8: note: forward declaration of 'struct
> adapter' struct adapter;
>
>
>
> Kind regards,
>
> oh
problem occurs when
WITH_OFED=YES
is set in /etc/src.conf.
More information about the svn-src-all
mailing list