From nobody Mon Jun 22 18:37:51 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gkcQ74t4pz6hnP9 for ; Mon, 22 Jun 2026 18:37:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gkcQ72lBcz3nfk for ; Mon, 22 Jun 2026 18:37:51 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1782153471; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fNnTZOvI5EvcvXFYb/wgjheutLSn777wtTyAAUdY5Ko=; b=U03uWSJxlvbvyjATsfEmevQWdlxof70hnTrLW77x/t4rqONNzJRCqXiTnQkL7dQRnNukmH 9hr6+v+1q1nOz8PyzOQoJ4KzEHFEjNujHVn125IJF+eW3du3AePhm6E5kCNsY//EcNkwtA uk8T9Fxmm7xTK29iL+2fNaLhZObtHaREIEP0gL+hD2LZwcRkWu/ZiLlHDvX3XcZXBJ7C4H K2eLC96E+4wOGsDvJN4+7nupKaGsRlEKtWpcnMs6dupzc0QIroUQt/rgStQjZvWFNKld98 2cgiDguTrs4Su/Gtuvad7cGEtfwIVQM/PADdZXJS6rGmk7j78UaX8feLLSCv1Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1782153471; a=rsa-sha256; cv=none; b=ubJiVmbt7ngwd9uW7HhgPO9IRODa0V254Ew/aGMTJIaqjKZl0fXVhbOGan8gRTxYtjI2xG C5qFRktFJpqEYF5A2PPTPUBIJnDKZ/3Ejt2M+GD8Wnj1/KiR9Cgt4yt3KCY0m/5cZj7By8 OoXY7SnhhTafI7ZUEf1fnIitPc4IVESpaBJ4UCtPaUw59R9EcKSYiXPhf5IQDrb/wOkc37 rE2MQf71oNByRAzQgsSWsof35+pWhLeLsf9idgYHVAXRCHMJ4MYQW4DyUGpraloP8r/5vn ALQz+7yEcJKmxSLesciBa9ghnpL64+iFGhDnJO0mJdhjA3dqAV5lyuDg4Fa31g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1782153471; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fNnTZOvI5EvcvXFYb/wgjheutLSn777wtTyAAUdY5Ko=; b=Vil3zitdkh258/tGggC/2jOfX50cllwtDYZltK1JvStVLd5DlN22nwYtIUS1sgX05Ljywo DoZLMCwKDoRYTiaSwQ4zfDpR7ZatkQ/xEoeFG1ujzXSHVWxNNTyQ/oVt6pBFKzQ0CUO+gH 7RPxQAazTcvJMv7jBc6+A4RC2vv6amMLbRHg/mp9VY/MHmErCIGb/Ekn0R6xFVpIgIScXt dXk1FhOkAyr0XmRP01pCEO0ZWkFsleJxRH4pDFrod0q+O7CBUJpDUa0nBO1pK2B9W8nSmA waCaPnBrUjusTf9E8g7I72f+hHcQTIqBoTXf+hl6wZILUKJ45Jxhw4fpR0jsSQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4gkcQ71b8Tz10PF for ; Mon, 22 Jun 2026 18:37:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3194c by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Mon, 22 Jun 2026 18:37:51 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 259a763e343d - main - OFED: Implement ib_process_cq_direct List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 259a763e343d36a6030e0bba5ae7ed05c123c2fe Auto-Submitted: auto-generated Date: Mon, 22 Jun 2026 18:37:51 +0000 Message-Id: <6a3980ff.3194c.3cec7756@gitrepo.freebsd.org> The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=259a763e343d36a6030e0bba5ae7ed05c123c2fe commit 259a763e343d36a6030e0bba5ae7ed05c123c2fe Author: John Baldwin AuthorDate: 2025-12-05 20:56:35 +0000 Commit: John Baldwin CommitDate: 2026-06-22 18:36:52 +0000 OFED: Implement ib_process_cq_direct This is largely pulled from the original Linux commit to add cq.c. Note that irq_poll is still not supported, but polling should now be possible whereas it wasn't really before. Tested by: Wafa Hamzah (mlx5_ib) Tested by: John Baldwin (iw_cxgbe) Obtained from: Linux commit 14d3a3b2498edadec344cb11e60e66091f5daf63 Sponsored by: Chelsio Communications --- sys/ofed/drivers/infiniband/core/ib_cq.c | 96 ++++++++++++++++++++++---------- sys/ofed/include/rdma/ib_verbs.h | 3 + 2 files changed, 70 insertions(+), 29 deletions(-) diff --git a/sys/ofed/drivers/infiniband/core/ib_cq.c b/sys/ofed/drivers/infiniband/core/ib_cq.c index d455f2c7c9ed..d6c897392022 100644 --- a/sys/ofed/drivers/infiniband/core/ib_cq.c +++ b/sys/ofed/drivers/infiniband/core/ib_cq.c @@ -39,42 +39,73 @@ #include -#define IB_CQ_POLL_MAX 16 -/* maximum number of completions per poll loop */ -#define IB_CQ_POLL_BUDGET 65536 -#define IB_CQ_POLL_FLAGS (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS) +/* # of WCs to poll for with a single call to ib_poll_cq */ +#define IB_POLL_BATCH 16 -static void -ib_cq_poll_work(struct work_struct *work) +/* # of WCs to iterate over before yielding */ +#define IB_POLL_BUDGET_WORKQUEUE 65536 + +#define IB_POLL_FLAGS \ + (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS) + +static int __ib_process_cq(struct ib_cq *cq, int budget) { - struct ib_wc ib_wc[IB_CQ_POLL_MAX]; - struct ib_cq *cq = container_of(work, struct ib_cq, work); - int total = 0; - int i; - int n; + int i, n, completed = 0; - while (1) { - n = ib_poll_cq(cq, IB_CQ_POLL_MAX, ib_wc); + while ((n = ib_poll_cq(cq, IB_POLL_BATCH, cq->wc)) > 0) { for (i = 0; i < n; i++) { - struct ib_wc *wc = ib_wc + i; + struct ib_wc *wc = &cq->wc[i]; - if (wc->wr_cqe != NULL) + if (wc->wr_cqe) wc->wr_cqe->done(cq, wc); - } - - if (n != IB_CQ_POLL_MAX) { - if (ib_req_notify_cq(cq, IB_CQ_POLL_FLAGS) > 0) - break; else - return; + WARN_ON_ONCE(wc->status == IB_WC_SUCCESS); } - total += n; - if (total >= IB_CQ_POLL_BUDGET) + + completed += n; + + if (n != IB_POLL_BATCH || + (budget != -1 && completed >= budget)) break; } - /* give other work structs a chance */ - queue_work(ib_comp_wq, &cq->work); + return completed; +} + +/** + * ib_process_direct_cq - process a CQ in caller context + * @cq: CQ to process + * @budget: number of CQEs to poll for + * + * This function is used to process all outstanding CQ entries on a + * %IB_POLL_DIRECT CQ. It does not offload CQ processing to a different + * context and does not ask for completion interrupts from the HCA. + * + * Note: for compatibility reasons -1 can be passed in %budget for unlimited + * polling. Do not use this feature in new code, it will be removed soon. + */ +int ib_process_cq_direct(struct ib_cq *cq, int budget) +{ + WARN_ON_ONCE(cq->poll_ctx != IB_POLL_DIRECT); + + return __ib_process_cq(cq, budget); +} +EXPORT_SYMBOL(ib_process_cq_direct); + +static void ib_cq_completion_direct(struct ib_cq *cq, void *private) +{ + WARN_ONCE(1, "got unsolicited completion for CQ 0x%p\n", cq); +} + +static void ib_cq_poll_work(struct work_struct *work) +{ + struct ib_cq *cq = container_of(work, struct ib_cq, work); + int completed; + + completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE); + if (completed >= IB_POLL_BUDGET_WORKQUEUE || + ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) + queue_work(ib_comp_wq, &cq->work); } static void @@ -94,7 +125,7 @@ __ib_alloc_cq_user(struct ib_device *dev, void *private, .comp_vector = comp_vector, }; struct ib_cq *cq; - int ret; + int ret = -ENOMEM; /* * Check for invalid parameters early on to avoid @@ -118,13 +149,17 @@ __ib_alloc_cq_user(struct ib_device *dev, void *private, cq->poll_ctx = poll_ctx; atomic_set(&cq->usecnt, 0); + cq->wc = kmalloc_array(IB_POLL_BATCH, sizeof(*cq->wc), GFP_KERNEL); + if (!cq->wc) + goto out_free_cq; + ret = dev->create_cq(cq, &cq_attr, NULL); if (ret) - goto out_free_cq; + goto out_free_wc; - switch (poll_ctx) { + switch (cq->poll_ctx) { case IB_POLL_DIRECT: - cq->comp_handler = NULL; /* no hardware completions */ + cq->comp_handler = ib_cq_completion_direct; break; case IB_POLL_SOFTIRQ: case IB_POLL_WORKQUEUE: @@ -137,6 +172,8 @@ __ib_alloc_cq_user(struct ib_device *dev, void *private, } return (cq); +out_free_wc: + kfree(cq->wc); out_free_cq: kfree(cq); return (ERR_PTR(ret)); @@ -161,6 +198,7 @@ ib_free_cq_user(struct ib_cq *cq, struct ib_udata *udata) break; } + kfree(cq->wc); cq->device->destroy_cq(cq, udata); kfree(cq); } diff --git a/sys/ofed/include/rdma/ib_verbs.h b/sys/ofed/include/rdma/ib_verbs.h index 086251183a55..8929cf2fe0b1 100644 --- a/sys/ofed/include/rdma/ib_verbs.h +++ b/sys/ofed/include/rdma/ib_verbs.h @@ -1472,6 +1472,7 @@ struct ib_cq { void *cq_context; int cqe; atomic_t usecnt; /* count number of work queues */ + struct ib_wc *wc; enum ib_poll_context poll_ctx; struct work_struct work; }; @@ -3354,6 +3355,8 @@ static inline void ib_free_cq(struct ib_cq *cq) ib_free_cq_user(cq, NULL); } +int ib_process_cq_direct(struct ib_cq *cq, int budget); + /** * ib_create_cq - Creates a CQ on the specified device. * @device: The device on which to create the CQ.