From nobody Fri Apr 29 23:12:39 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2B6841ABD02A; Fri, 29 Apr 2022 23:12:41 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KqpDS2J4rz3wK0; Fri, 29 Apr 2022 23:12:40 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1651273960; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UUq2G7Et2dopb+n+M1Fxbp2db8FRUaE2UlxGUp8shos=; b=ady2YJcsoYWP4qZXU/tPtItnWh3nup9GRYmJDWofJ7H0SbpJE2vbYraiVk4esIrJqT3H9Q zTb5+yGzfnkfjbc1EKgHi//QXDqJvjdGQr6CtLcMfqsDpw7j+kXq1+pCqd1PAmcQ0pp7Rl NQZWP/gdOLh6kgE6djtMuacpsEtJYvB04KwG/JpBKcjwxdXEgm0ADBFgk/ZfBe9NnP3OH5 CPbBA4/hfm6GyfQcECtl4h6w1LTvlAE4EDEbQHqVUE0R6NZhR0wADOBQqwPOhAIz7WEycC n10V5+phfxCeX9lEgojQQoCVlnAOQ1U2e7nnjr8aHa4V9c+YuK3cLIqArwyaQw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CF20615FC1; Fri, 29 Apr 2022 23:12:39 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 23TNCdOi044947; Fri, 29 Apr 2022 23:12:39 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 23TNCduL044946; Fri, 29 Apr 2022 23:12:39 GMT (envelope-from git) Date: Fri, 29 Apr 2022 23:12:39 GMT Message-Id: <202204292312.23TNCduL044946@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: John Baldwin Subject: git: 05d87fdb32bf - stable/13 - cxgbei: Rework parsing of pre-offload PDUs. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 05d87fdb32bf6b2bed35c077be24f1a4854a5862 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1651273960; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UUq2G7Et2dopb+n+M1Fxbp2db8FRUaE2UlxGUp8shos=; b=BvemwphCtweHpl4uTWM2qxtXLGHaLkOq1OH+EggiwUS659Q3XZwAXW7rWQyttVeTwPZFNh t5sb/59XKwH9RAs9WloiHf0VUTHgBfIPIfbYOPBgaPHIttSZ6PGvs11LBMJsvLMO2QpycB UQmsgDnP1M911p+YzsPTpuxyGW4H/3UawdJVi/gwkkfVyhGh7hxlLX4YmEsFaLAmftPu1M dsLsYf5ibLWLihdGoooMFkcKryvq3Kz31jXhEMXnhAAUhOatcl4xZSLuiFYFl9w6zcer7W G1fUA7g7Jf+iW+eSFFx96a4K4hhopxNK4oKCrX+Fp62GC17PE9QqwXVwUkreBw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1651273960; a=rsa-sha256; cv=none; b=FCgd9U7RIVFyOm32D1l+HfQgn3dsWQsAEnI1f4u/BMhMcVyA3e5mVsaV+K3/1tK7X8reUX 5/b+eFG4fdwE/z5VaT5DVQLLx4tewlbOZ5ybF9hHydqa1c6x+BzhXC92xdVk20aALDDnxs YKZgRtP8MHh/cHaiiIu68+VGTCvWz/y5F56ka6W5yhf1+uB8mJotND1EGULJSfrE2gTrLB fhjyGb8ZSJZXK3h3xCojryEqvGF9RPptJApHLCsbs3Gd56zKz7dDsLX+Ok8zJMKech9VW8 x39CMOpogonjp5ZZbhZQfRto+zHMh2rG1WQBPlv32SuUuYI1i3fVNN2e6ym0KA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=05d87fdb32bf6b2bed35c077be24f1a4854a5862 commit 05d87fdb32bf6b2bed35c077be24f1a4854a5862 Author: John Baldwin AuthorDate: 2022-02-04 23:38:49 +0000 Commit: John Baldwin CommitDate: 2022-04-29 23:06:48 +0000 cxgbei: Rework parsing of pre-offload PDUs. sbcut() returns mbufs in reverse order so is not suitable for reading data from the socket buffer. Instead, check for already-received data in the receive worker thread before passing offload PDUs up to the iSCSI layer. This uses soreceive() to read data from the socket and is also to use M_WAITOK since it now runs from a worker thread instead of an interrupt thread. Also, fix decoding of the data segment length for pre-offload PDUs. Reported by: Jithesh Arakkan @ Chelsio Fixes: a8c4147edcdc cxgbei: Parse all PDUs received prior to enabling offload mode. Sponsored by: Chelsio Communications (cherry picked from commit 74fea8eb4fee65163fa745d0dbfcefc138ff7925) --- sys/dev/cxgbe/cxgbei/cxgbei.c | 240 ++++++++++++++++++++++-------------------- 1 file changed, 124 insertions(+), 116 deletions(-) diff --git a/sys/dev/cxgbe/cxgbei/cxgbei.c b/sys/dev/cxgbe/cxgbei/cxgbei.c index bca21d211abd..4a8df99b3d48 100644 --- a/sys/dev/cxgbe/cxgbei/cxgbei.c +++ b/sys/dev/cxgbe/cxgbei/cxgbei.c @@ -52,6 +52,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -309,79 +310,95 @@ mbuf_crc32c_helper(void *arg, void *data, u_int len) return (0); } -static bool -parse_pdus(struct toepcb *toep, struct icl_cxgbei_conn *icc, struct sockbuf *sb) +static struct icl_pdu * +parse_pdu(struct socket *so, struct toepcb *toep, struct icl_cxgbei_conn *icc, + struct sockbuf *sb, u_int total_len) { + struct uio uio; + struct iovec iov[2]; struct iscsi_bhs bhs; struct mbuf *m; struct icl_pdu *ip; - u_int ahs_len, data_len, header_len, pdu_len, total_len; + u_int ahs_len, data_len, header_len, pdu_len; uint32_t calc_digest, wire_digest; + int error; - total_len = sbused(sb); - CTR3(KTR_CXGBE, "%s: tid %u, %u bytes in so_rcv", __func__, toep->tid, - total_len); - - m = sbcut_locked(sb, total_len); - KASSERT(m_length(m, NULL) == total_len, - ("sbcut returned less data (%u vs %u)", total_len, - m_length(m, NULL))); + uio.uio_segflg = UIO_SYSSPACE; + uio.uio_rw = UIO_READ; + uio.uio_td = curthread; header_len = sizeof(struct iscsi_bhs); if (icc->ic.ic_header_crc32c) header_len += ISCSI_HEADER_DIGEST_SIZE; - for (;;) { - if (total_len < sizeof(struct iscsi_bhs)) { - ICL_WARN("truncated pre-offload PDU with len %u", - total_len); - m_freem(m); - return (false); - } - m_copydata(m, 0, sizeof(struct iscsi_bhs), (caddr_t)&bhs); - - ahs_len = bhs.bhs_total_ahs_len * 4; - data_len = bhs.bhs_data_segment_len[0] << 16 | - bhs.bhs_data_segment_len[1] << 8 | - bhs.bhs_data_segment_len[0]; - pdu_len = header_len + ahs_len + roundup2(data_len, 4); - if (icc->ic.ic_data_crc32c && data_len != 0) - pdu_len += ISCSI_DATA_DIGEST_SIZE; - - if (total_len < pdu_len) { - ICL_WARN("truncated pre-offload PDU len %u vs %u", - total_len, pdu_len); - m_freem(m); - return (false); - } - if (ahs_len != 0) { - ICL_WARN("received pre-offload PDU with AHS"); - m_freem(m); - return (false); - } + if (total_len < header_len) { + ICL_WARN("truncated pre-offload PDU with len %u", total_len); + return (NULL); + } - if (icc->ic.ic_header_crc32c) { - m_copydata(m, sizeof(struct iscsi_bhs), - sizeof(wire_digest), (caddr_t)&wire_digest); + iov[0].iov_base = &bhs; + iov[0].iov_len = sizeof(bhs); + iov[1].iov_base = &wire_digest; + iov[1].iov_len = sizeof(wire_digest); + uio.uio_iov = iov; + uio.uio_iovcnt = 1; + uio.uio_offset = 0; + uio.uio_resid = header_len; + error = soreceive(so, NULL, &uio, NULL, NULL, NULL); + if (error != 0) { + ICL_WARN("failed to read BHS from pre-offload PDU: %d", error); + return (NULL); + } - calc_digest = calculate_crc32c(0xffffffff, - (caddr_t)&bhs, sizeof(bhs)); - calc_digest ^= 0xffffffff; - if (calc_digest != wire_digest) { - ICL_WARN("received pre-offload PDU 0x%02x " - "with invalid header digest (0x%x vs 0x%x)", - bhs.bhs_opcode, wire_digest, calc_digest); - toep->ofld_rxq->rx_iscsi_header_digest_errors++; - m_free(m); - return (false); - } + ahs_len = bhs.bhs_total_ahs_len * 4; + data_len = bhs.bhs_data_segment_len[0] << 16 | + bhs.bhs_data_segment_len[1] << 8 | + bhs.bhs_data_segment_len[2]; + pdu_len = header_len + ahs_len + roundup2(data_len, 4); + if (icc->ic.ic_data_crc32c && data_len != 0) + pdu_len += ISCSI_DATA_DIGEST_SIZE; + + if (total_len < pdu_len) { + ICL_WARN("truncated pre-offload PDU len %u vs %u", total_len, + pdu_len); + return (NULL); + } + + if (ahs_len != 0) { + ICL_WARN("received pre-offload PDU with AHS"); + return (NULL); + } + + if (icc->ic.ic_header_crc32c) { + calc_digest = calculate_crc32c(0xffffffff, (caddr_t)&bhs, + sizeof(bhs)); + calc_digest ^= 0xffffffff; + if (calc_digest != wire_digest) { + ICL_WARN("received pre-offload PDU 0x%02x with " + "invalid header digest (0x%x vs 0x%x)", + bhs.bhs_opcode, wire_digest, calc_digest); + toep->ofld_rxq->rx_iscsi_header_digest_errors++; + return (NULL); } + } - m_adj(m, header_len); + m = NULL; + if (data_len != 0) { + uio.uio_iov = NULL; + uio.uio_resid = roundup2(data_len, 4); + if (icc->ic.ic_data_crc32c) + uio.uio_resid += ISCSI_DATA_DIGEST_SIZE; + + error = soreceive(so, NULL, &uio, &m, NULL, NULL); + if (error != 0) { + ICL_WARN("failed to read data payload from " + "pre-offload PDU: %d", error); + return (NULL); + } - if (icc->ic.ic_data_crc32c && data_len != 0) { - m_copydata(m, data_len, sizeof(wire_digest), - (caddr_t)&wire_digest); + if (icc->ic.ic_data_crc32c) { + m_copydata(m, roundup2(data_len, 4), + sizeof(wire_digest), (caddr_t)&wire_digest); calc_digest = 0xffffffff; m_apply(m, 0, roundup2(data_len, 4), mbuf_crc32c_helper, @@ -392,41 +409,55 @@ parse_pdus(struct toepcb *toep, struct icl_cxgbei_conn *icc, struct sockbuf *sb) "with invalid data digest (0x%x vs 0x%x)", bhs.bhs_opcode, wire_digest, calc_digest); toep->ofld_rxq->rx_iscsi_data_digest_errors++; - m_free(m); - return (false); + m_freem(m); + return (NULL); } } + } - ip = icl_cxgbei_new_pdu(M_NOWAIT); - if (ip == NULL) - CXGBE_UNIMPLEMENTED("PDU allocation failure"); - icl_cxgbei_new_pdu_set_conn(ip, &icc->ic); - *ip->ip_bhs = bhs; - ip->ip_data_len = data_len; - if (data_len != 0) - ip->ip_data_mbuf = m; + ip = icl_cxgbei_new_pdu(M_WAITOK); + icl_cxgbei_new_pdu_set_conn(ip, &icc->ic); + *ip->ip_bhs = bhs; + ip->ip_data_len = data_len; + ip->ip_data_mbuf = m; + return (ip); +} - STAILQ_INSERT_TAIL(&icc->rcvd_pdus, ip, ip_next); +static void +parse_pdus(struct icl_cxgbei_conn *icc, struct sockbuf *sb) +{ + struct icl_conn *ic = &icc->ic; + struct socket *so = ic->ic_socket; + struct toepcb *toep = icc->toep; + struct icl_pdu *ip, *lastip; + u_int total_len; - total_len -= pdu_len; - if (total_len == 0) { - if (data_len == 0) - m_freem(m); - return (true); - } + SOCKBUF_LOCK_ASSERT(sb); - if (data_len != 0) { - m = m_split(m, roundup2(data_len, 4), M_NOWAIT); - if (m == NULL) { - ICL_WARN("failed to split mbuf chain for " - "pre-offload PDU"); + CTR3(KTR_CXGBE, "%s: tid %u, %u bytes in so_rcv", __func__, toep->tid, + sbused(sb)); - /* Don't free the mbuf chain as 'ip' owns it. */ - return (false); - } - if (icc->ic.ic_data_crc32c) - m_adj(m, ISCSI_DATA_DIGEST_SIZE); + lastip = NULL; + while (sbused(sb) != 0 && (sb->sb_state & SBS_CANTRCVMORE) == 0) { + total_len = sbused(sb); + SOCKBUF_UNLOCK(sb); + + ip = parse_pdu(so, toep, icc, sb, total_len); + + if (ip == NULL) { + ic->ic_error(ic); + SOCKBUF_LOCK(sb); + return; } + + if (lastip == NULL) + STAILQ_INSERT_HEAD(&icc->rcvd_pdus, ip, ip_next); + else + STAILQ_INSERT_AFTER(&icc->rcvd_pdus, lastip, ip, + ip_next); + lastip = ip; + + SOCKBUF_LOCK(sb); } } @@ -551,22 +582,6 @@ do_rx_iscsi_ddp(struct sge_iq *iq, const struct rss_header *rss, struct mbuf *m) return (0); } - if (__predict_false(sbused(sb)) != 0) { - /* - * PDUs were received before the tid transitioned to ULP mode. - * Convert them to icl_cxgbei_pdus and send them to ICL before - * the PDU in icp/ip. - */ - if (!parse_pdus(toep, icc, sb)) { - SOCKBUF_UNLOCK(sb); - INP_WUNLOCK(inp); - - icl_cxgbei_conn_pdu_free(NULL, ip); - toep->ulpcb2 = NULL; - ic->ic_error(ic); - return (0); - } - } icl_cxgbei_new_pdu_set_conn(ip, ic); STAILQ_INSERT_TAIL(&icc->rcvd_pdus, ip, ip_next); @@ -817,22 +832,6 @@ do_rx_iscsi_cmp(struct sge_iq *iq, const struct rss_header *rss, struct mbuf *m) return (0); } - if (__predict_false(sbused(sb)) != 0) { - /* - * PDUs were received before the tid transitioned to ULP mode. - * Convert them to icl_cxgbei_pdus and send them to ICL before - * the PDU in icp/ip. - */ - if (!parse_pdus(toep, icc, sb)) { - SOCKBUF_UNLOCK(sb); - INP_WUNLOCK(inp); - - icl_cxgbei_conn_pdu_free(NULL, ip); - toep->ulpcb2 = NULL; - ic->ic_error(ic); - return (0); - } - } icl_cxgbei_new_pdu_set_conn(ip, ic); /* Enqueue the PDU to the received pdus queue. */ @@ -971,6 +970,15 @@ cwt_main(void *arg) sb = &ic->ic_socket->so_rcv; SOCKBUF_LOCK(sb); + if (__predict_false(sbused(sb)) != 0) { + /* + * PDUs were received before the tid + * transitioned to ULP mode. Convert + * them to icl_cxgbei_pdus and insert + * them into the head of rcvd_pdus. + */ + parse_pdus(icc, sb); + } MPASS(icc->rx_flags & RXF_ACTIVE); if (__predict_true(!(sb->sb_state & SBS_CANTRCVMORE))) { MPASS(STAILQ_EMPTY(&rx_pdus));