From nobody Fri Oct 29 23:58:32 2021 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D565D181DB15; Fri, 29 Oct 2021 23:58:36 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HgzsR4ZzRz4VXw; Fri, 29 Oct 2021 23:58:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7779C20070; Fri, 29 Oct 2021 23:58:32 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 19TNwWK1003900; Fri, 29 Oct 2021 23:58:32 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 19TNwW3a003899; Fri, 29 Oct 2021 23:58:32 GMT (envelope-from git) Date: Fri, 29 Oct 2021 23:58:32 GMT Message-Id: <202110292358.19TNwW3a003899@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: John Baldwin Subject: git: 37c5daa85596 - stable/13 - cxgbei: Wait for the final CPL to be received in icl_cxgbei_conn_close. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 37c5daa855966118e99bca7f5c79f7c20c61529c Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=37c5daa855966118e99bca7f5c79f7c20c61529c commit 37c5daa855966118e99bca7f5c79f7c20c61529c Author: John Baldwin AuthorDate: 2021-08-12 15:48:14 +0000 Commit: John Baldwin CommitDate: 2021-10-29 23:27:31 +0000 cxgbei: Wait for the final CPL to be received in icl_cxgbei_conn_close. A socket in the FIN_WAIT_1 state is marked disconnected by do_close_con_rpl() even though there might still receive data pending. This is because the socket at that point has set SBS_CANTRCVMORE which causes the protocol layer to discard any data received before the FIN. However, icl_cxgbei_conn_close needs to wait until all the data has been discarded. Replace the wait for SS_ISDISCONNECTED with instead waiting for final_cpl_received() to be called. Reported by: Jithesh Arakkan @ Chelsio Sponsored by: Chelsio Communications (cherry picked from commit 2eb0e53a6b5ec1a72be70e966d4e562e1a8d4e88) --- sys/dev/cxgbe/cxgbei/icl_cxgbei.c | 28 +++++++++++++++++++++------- sys/dev/cxgbe/tom/t4_tom.c | 12 +++++++++++- sys/dev/cxgbe/tom/t4_tom.h | 1 + 3 files changed, 33 insertions(+), 8 deletions(-) diff --git a/sys/dev/cxgbe/cxgbei/icl_cxgbei.c b/sys/dev/cxgbe/cxgbei/icl_cxgbei.c index a57d26ae21b8..5526388915f7 100644 --- a/sys/dev/cxgbe/cxgbei/icl_cxgbei.c +++ b/sys/dev/cxgbe/cxgbei/icl_cxgbei.c @@ -947,6 +947,18 @@ icl_cxgbei_conn_close(struct icl_conn *ic) icl_cxgbei_pdu_done(ip, ENOTCONN); } SOCKBUF_UNLOCK(sb); + + /* + * Grab a reference to use when waiting for the final + * CPL to be received. If toep->inp is NULL, then + * final_cpl_received() has already been called (e.g. + * due to the peer sending a RST). + */ + if (toep->inp != NULL) { + toep = hold_toepcb(toep); + toep->flags |= TPF_WAITING_FOR_FINAL; + } else + toep = NULL; } INP_WUNLOCK(inp); @@ -959,7 +971,6 @@ icl_cxgbei_conn_close(struct icl_conn *ic) * queues were purged instead of delivered reliably but soabort isn't * really general purpose and wouldn't do the right thing here. */ - soref(so); soclose(so); /* @@ -969,12 +980,15 @@ icl_cxgbei_conn_close(struct icl_conn *ic) * Callers assume that it is safe to free buffers for tasks * and transfers after this function returns. */ - SOCK_LOCK(so); - while ((so->so_state & SS_ISDISCONNECTED) == 0) - mtx_sleep(&so->so_timeo, SOCK_MTX(so), PSOCK, "conclo2", 0); - CURVNET_SET(so->so_vnet); - sorele(so); - CURVNET_RESTORE(); + if (toep != NULL) { + struct mtx *lock = mtx_pool_find(mtxpool_sleep, toep); + + mtx_lock(lock); + while ((toep->flags & TPF_WAITING_FOR_FINAL) != 0) + mtx_sleep(toep, lock, PSOCK, "conclo2", 0); + mtx_unlock(lock); + free_toepcb(toep); + } } static void diff --git a/sys/dev/cxgbe/tom/t4_tom.c b/sys/dev/cxgbe/tom/t4_tom.c index f3cd8270d24d..fdd0ab870f43 100644 --- a/sys/dev/cxgbe/tom/t4_tom.c +++ b/sys/dev/cxgbe/tom/t4_tom.c @@ -848,6 +848,7 @@ void final_cpl_received(struct toepcb *toep) { struct inpcb *inp = toep->inp; + bool need_wakeup; KASSERT(inp != NULL, ("%s: inp is NULL", __func__)); INP_WLOCK_ASSERT(inp); @@ -862,7 +863,8 @@ final_cpl_received(struct toepcb *toep) else if (ulp_mode(toep) == ULP_MODE_TLS) tls_detach(toep); toep->inp = NULL; - toep->flags &= ~TPF_CPL_PENDING; + need_wakeup = (toep->flags & TPF_WAITING_FOR_FINAL) != 0; + toep->flags &= ~(TPF_CPL_PENDING | TPF_WAITING_FOR_FINAL); mbufq_drain(&toep->ulp_pduq); mbufq_drain(&toep->ulp_pdu_reclaimq); @@ -871,6 +873,14 @@ final_cpl_received(struct toepcb *toep) if (!in_pcbrele_wlocked(inp)) INP_WUNLOCK(inp); + + if (need_wakeup) { + struct mtx *lock = mtx_pool_find(mtxpool_sleep, toep); + + mtx_lock(lock); + wakeup(toep); + mtx_unlock(lock); + } } void diff --git a/sys/dev/cxgbe/tom/t4_tom.h b/sys/dev/cxgbe/tom/t4_tom.h index 89c38f8c988a..2224ece20c38 100644 --- a/sys/dev/cxgbe/tom/t4_tom.h +++ b/sys/dev/cxgbe/tom/t4_tom.h @@ -76,6 +76,7 @@ enum { TPF_INITIALIZED = (1 << 12), /* init_toepcb has been called */ TPF_TLS_RECEIVE = (1 << 13), /* should receive TLS records */ TPF_TLS_ESTABLISHED = (1 << 14), /* TLS handshake timer initialized */ + TPF_WAITING_FOR_FINAL = (1<< 15), /* waiting for wakeup on final CPL */ }; enum {