From nobody Tue Sep 15 14:47:48 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hklHT0Vlmz6sK43 for ; Tue, 15 Sep 2026 14:47:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hklHS5smgz4Nnm for ; Tue, 15 Sep 2026 14:47:48 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1789483668; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cc0v+1qhYqg12sEnw3PpgRRT4BkD6KSF+6F8Qr2IxCQ=; b=ezOOZghRBT/o/8K2VakW3lknxN+TZcjUOODNGitmWqSaTRHlX6n/V0hQ1x6OkPBo1+lWra EXAFiFmuTXCJqcB0xThAe2bGS+JWV3Q4TqzpAkDjWRIypxMalV0isEXNe6r9bl4pZR34q7 4MVhU0B9y2vJ2GBQMws64xL234Hbvd5kmieftrhQp0t5ohpdIqatuGJQtnThgawD551jub xLkbqI/rrGXfSisKjFzc2vVS7Fyb3X0YusedK2a2wDi52wCluVpYImLz9po/LlRVSHwHkZ KZeWnWm4OcymnAGmZxVqYzDUj+nhokyF3W/RtjCFTqFE1oPUgH9btGrb2xqmmg== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1789483668; b=oCYiLvOisJCO9/goXh73ICZ3+ESgO77DbBmca4AFfWzC9bU0ASCTiA92QptsQoyZtJGGRb MLAE5QvVWxpjbhAuAALScz6M5zbefDhAOeNFIxtykpwjUtbI78OYSERpkXhlhwAc1lt/s9 e753iiGAACmVJ1aKUdOK8hdDhbyH4vdI0bmE5RV94ewRbIcTehAXrfy3wuY9SFzjuoEVkd Cp6aLP+5gGuaE2zaRrYn3bBYLGGQGxdfXkIaoeEncbbhw483zwwuBYTeccgFBMgOd3wumP 8aIYuxRK4VZflcjpFxyx9yVmQgRX1hPkWiUiKTznS3doTuU+14MjW2d4s6jMQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1789483668; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cc0v+1qhYqg12sEnw3PpgRRT4BkD6KSF+6F8Qr2IxCQ=; b=w7f/W77rxnx7+HfrLemc4WhXpsODgqiL64DAIoCB/aJD2hDmC3UWUs0QMf2CSxlxKhUOl6 yGZ4MMJNHFb04tPrWGrzQN47X4jM0vwFcl5Kp9lKbBpm5YYNXfivtDBSWYGi0Apc8BBJ65 STNWvHetGjb1FopSPvgjqFCkqvYwnlq9ZrOfRnFa45YoHEhEGyZtGzmeDZ9wVwclU9IoJj kW+Dh+YrGEbwX099oqYa3rCiLaF9Bc9awCdfN2zdtdxDtCvZSdarSxeXYQsGlK/spm4HIx rN+Kdoo95h0f+qG9MZ7OLXVHWTgOne5nW/9S/mVZPP1S7cLCNwYvsi9IKcKKuQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hklHS38vYz14Ph for ; Tue, 15 Sep 2026 14:47:48 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 37f07 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Tue, 15 Sep 2026 14:47:48 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Cc: Potnuri Bharat Teja From: John Baldwin Subject: git: 6262cbed3c37 - main - libcxgb4: Support send/recv drain work requests List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6262cbed3c37aff6ee9ff4dad7b14147bb40ac7e Auto-Submitted: auto-generated Date: Tue, 15 Sep 2026 14:47:48 +0000 Message-Id: <6aa95a94.37f07.56214338@gitrepo.freebsd.org> The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=6262cbed3c37aff6ee9ff4dad7b14147bb40ac7e commit 6262cbed3c37aff6ee9ff4dad7b14147bb40ac7e Author: Potnuri Bharat Teja AuthorDate: 2018-12-17 18:09:41 +0000 Commit: John Baldwin CommitDate: 2026-09-15 14:22:39 +0000 libcxgb4: Support send/recv drain work requests This change allows libcxgb4 to post send/recv drain work requests after a QP is flushed and schedules a CQE on the software CQ for the application to note the drain's completion. Sponsored by: Chelsio Communications --- contrib/ofed/libcxgb4/cq.c | 15 ++++++ contrib/ofed/libcxgb4/qp.c | 132 ++++++++++++++++++++++++++++++++++++++++++--- contrib/ofed/libcxgb4/t4.h | 14 +++++ 3 files changed, 155 insertions(+), 6 deletions(-) diff --git a/contrib/ofed/libcxgb4/cq.c b/contrib/ofed/libcxgb4/cq.c index a5ce335d7e60..b03a52a59159 100644 --- a/contrib/ofed/libcxgb4/cq.c +++ b/contrib/ofed/libcxgb4/cq.c @@ -283,6 +283,12 @@ next_cqe: static int cqe_completes_wr(struct t4_cqe *cqe, struct t4_wq *wq) { + if (DRAIN_CQE(cqe)) { + fprintf(stderr, "Unexpected DRAIN CQE qp id %u!\n", + wq->sq.qid); + return 0; + } + if (CQE_OPCODE(cqe) == FW_RI_TERMINATE) return 0; @@ -370,6 +376,15 @@ static int poll_cq(struct t4_wq *wq, struct t4_cq *cq, struct t4_cqe *cqe, goto skip_cqe; } + /* + * Special cqe for drain WR completions... + */ + if (DRAIN_CQE(hw_cqe)) { + *cookie = CQE_DRAIN_COOKIE(hw_cqe); + *cqe = *hw_cqe; + goto skip_cqe; + } + /* * Gotta tweak READ completions: * 1) the cqe doesn't contain the sq_wptr from the wr. diff --git a/contrib/ofed/libcxgb4/qp.c b/contrib/ofed/libcxgb4/qp.c index 7f77de77ae9d..a41381bf4907 100644 --- a/contrib/ofed/libcxgb4/qp.c +++ b/contrib/ofed/libcxgb4/qp.c @@ -396,6 +396,116 @@ static void ring_kernel_db(struct c4iw_qp *qhp, u32 qid, u16 idx) assert(!ret); } +static int ibv_to_fw_opcode(int ib_opcode) +{ + int opcode; + + switch (ib_opcode) { + case IBV_WR_SEND_WITH_INV: + opcode = FW_RI_SEND_WITH_INV; + break; + case IBV_WR_SEND: + opcode = FW_RI_SEND; + break; + case IBV_WR_RDMA_WRITE: + opcode = FW_RI_RDMA_WRITE; + break; + case IBV_WR_RDMA_WRITE_WITH_IMM: + opcode = FW_RI_WRITE_IMMEDIATE; + break; + case IBV_WR_RDMA_READ: + opcode = FW_RI_READ_REQ; + break; + default: + opcode = -EINVAL; + } + return opcode; +} + +static int complete_sq_drain_wr(struct c4iw_qp *qhp, struct ibv_send_wr *wr) +{ + struct t4_cqe cqe = {}; + struct c4iw_cq *schp; + struct t4_cq *cq; + int opcode; + + schp = to_c4iw_cq(qhp->ibv_qp.send_cq); + cq = &schp->cq; + + opcode = ibv_to_fw_opcode(wr->opcode); + if (opcode < 0) + return opcode; + + PDBG("drain sq id %u\n", qhp->wq.sq.qid); + cqe.u.drain_cookie = wr->wr_id; + cqe.header = htobe32(V_CQE_STATUS(T4_ERR_SWFLUSH) | + V_CQE_OPCODE(opcode) | + V_CQE_TYPE(1) | + V_CQE_SWCQE(1) | + V_CQE_DRAIN(1) | + V_CQE_QPID(qhp->wq.sq.qid)); + + pthread_spin_lock(&schp->lock); + cqe.bits_type_ts = htobe64(V_CQE_GENBIT((u64)cq->gen)); + cq->sw_queue[cq->sw_pidx] = cqe; + t4_swcq_produce(cq); + pthread_spin_unlock(&schp->lock); + + t4_clear_cq_armed(&schp->cq); + return 0; +} + +static int complete_sq_drain_wrs(struct c4iw_qp *qhp, struct ibv_send_wr *wr, + struct ibv_send_wr **bad_wr) +{ + int ret = 0; + + while (wr) { + ret = complete_sq_drain_wr(qhp, wr); + if (ret) { + *bad_wr = wr; + break; + } + wr = wr->next; + } + return ret; +} + +static void complete_rq_drain_wr(struct c4iw_qp *qhp, struct ibv_recv_wr *wr) +{ + struct t4_cqe cqe = {}; + struct c4iw_cq *rchp; + struct t4_cq *cq; + + rchp = to_c4iw_cq(qhp->ibv_qp.recv_cq); + cq = &rchp->cq; + + PDBG("drain rq id %u\n", qhp->wq.sq.qid); + cqe.u.drain_cookie = wr->wr_id; + cqe.header = htobe32(V_CQE_STATUS(T4_ERR_SWFLUSH) | + V_CQE_OPCODE(FW_RI_SEND) | + V_CQE_TYPE(0) | + V_CQE_SWCQE(1) | + V_CQE_DRAIN(1) | + V_CQE_QPID(qhp->wq.sq.qid)); + + pthread_spin_lock(&rchp->lock); + cqe.bits_type_ts = htobe64(V_CQE_GENBIT((u64)cq->gen)); + cq->sw_queue[cq->sw_pidx] = cqe; + t4_swcq_produce(cq); + pthread_spin_unlock(&rchp->lock); + + t4_clear_cq_armed(&rchp->cq); +} + +static void complete_rq_drain_wrs(struct c4iw_qp *qhp, struct ibv_recv_wr *wr) +{ + while (wr) { + complete_rq_drain_wr(qhp, wr); + wr = wr->next; + } +} + static void post_write_cmpl(struct c4iw_qp *qhp, struct ibv_send_wr *wr) { bool send_signaled = (wr->next->send_flags & IBV_SEND_SIGNALED) || @@ -472,10 +582,15 @@ int c4iw_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, qhp = to_c4iw_qp(ibqp); pthread_spin_lock(&qhp->lock); - if (t4_wq_in_error(&qhp->wq)) { + + /* + * If the qp has been flushed, then just insert a special + * drain cqe. + */ + if (qhp->wq.flushed) { pthread_spin_unlock(&qhp->lock); - *bad_wr = wr; - return -EINVAL; + err = complete_sq_drain_wrs(qhp, wr, bad_wr); + return err; } num_wrs = t4_sq_avail(&qhp->wq); if (num_wrs == 0) { @@ -612,10 +727,15 @@ int c4iw_post_receive(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, qhp = to_c4iw_qp(ibqp); pthread_spin_lock(&qhp->lock); - if (t4_wq_in_error(&qhp->wq)) { + + /* + * If the qp has been flushed, then just insert a special + * drain cqe. + */ + if (qhp->wq.flushed) { pthread_spin_unlock(&qhp->lock); - *bad_wr = wr; - return -EINVAL; + complete_rq_drain_wrs(qhp, wr); + return err; } INC_STAT(recv); num_wrs = t4_rq_avail(&qhp->wq); diff --git a/contrib/ofed/libcxgb4/t4.h b/contrib/ofed/libcxgb4/t4.h index 8027fdd39f8c..34f8db74e4e9 100644 --- a/contrib/ofed/libcxgb4/t4.h +++ b/contrib/ofed/libcxgb4/t4.h @@ -100,6 +100,7 @@ struct t4_status_page { __be16 pidx; u8 qp_err; /* flit 1 - sw owns */ u8 db_off; + u8 cq_armed; u8 pad; u16 host_wq_pidx; u16 host_cidx; @@ -217,6 +218,7 @@ struct t4_cqe { __be32 wrid_hi; __be32 wrid_low; } gen; + __u64 drain_cookie; struct { __be32 mo; __be32 msn; @@ -239,6 +241,11 @@ struct t4_cqe { #define G_CQE_SWCQE(x) ((((x) >> S_CQE_SWCQE)) & M_CQE_SWCQE) #define V_CQE_SWCQE(x) ((x)<> S_CQE_DRAIN)) & M_CQE_DRAIN) +#define V_CQE_DRAIN(x) ((x)<> S_CQE_STATUS)) & M_CQE_STATUS) @@ -255,6 +262,7 @@ struct t4_cqe { #define V_CQE_OPCODE(x) ((x)<header))) +#define DRAIN_CQE(x) (G_CQE_DRAIN(be32toh((x)->header))) #define CQE_QPID(x) (G_CQE_QPID(be32toh((x)->header))) #define CQE_TYPE(x) (G_CQE_TYPE(be32toh((x)->header))) #define SQ_TYPE(x) (CQE_TYPE((x))) @@ -281,6 +289,7 @@ struct t4_cqe { /* generic accessor macros */ #define CQE_WRID_HI(x) ((x)->u.gen.wrid_hi) #define CQE_WRID_LOW(x) ((x)->u.gen.wrid_low) +#define CQE_DRAIN_COOKIE(x) ((x)->u.drain_cookie) /* macros for flit 3 of the cqe */ #define S_CQE_GENBIT 63 @@ -604,6 +613,11 @@ struct t4_cq { u8 error; }; +static inline void t4_clear_cq_armed(struct t4_cq *cq) +{ + ((struct t4_status_page *)&cq->queue[cq->size])->cq_armed = 0; +} + static inline int t4_arm_cq(struct t4_cq *cq, int se) { u32 val;