From nobody Fri Oct 09 10:12:33 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4j1N2p0rNzz6vR44 for ; Fri, 09 Oct 2026 10:12:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4j1N2n60BRz4Xtf for ; Fri, 09 Oct 2026 10:12:33 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1791540753; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=pmrQXpO04s5w4pu8FCEZISoi2E1afKc88b+8vgV8Uxw=; b=wq57V8dgsn1dAtN2nMU9xgxMxd81im003q1Nx1IM8wQQZ8rCiicpNpI7KlFS/h2sWdC1zO +j+DiEaP4ygLZtGEATG+dpcvSCO1ess+yCW9rOXzkev//TbL3O+4bPYJloFj0iOoxrV1Nq Sb6ar5npCWk27r6PxIA+X0QeU3ol9PFbkPa1sbosRCGhznmWXGLZjUwPCj2O5fscRmmmXG 0AarHMuPctuk8E90BVGR0kZ7L/C4wvEJ/c88m46/rcNLCNvnHYwJqF7bl16a0PhQ/LvLcx YMKZWV4JaY0dF8YDeJAyZX/uZj4UpD8/TbBoGwFqBWE7yD0ocVmR8HgyeR5/NA== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1791540753; b=rUlXT9w7t2H0+1UKVnROUxJREQHgSHwwwUKC0N+6Xc73WZD+wsxTP/xO/jc6aYWIYqAU+1 SNYJ/xNuxyUwgOZwtiWE3MPmrEuMVpohnLQglv3g8rEkPJXe2msjYOnh2mXwIKH51i4dSM u2XiLPVK/HhHWGC+dxM41DX2rj+8tr1zuU2BWnThC9KJ23A5n1KR1gAMZcIPGRNwSXdlcD UVe0YVtEB5QCCRIV6xv5uGqNe9U53KGEpiZjEJn/jeMhOaRHFIjb9K84sVWYiCUl847aaR W/osz6bSjJX+YZPltJNt4tRJ/yqYSmWoYBlUCI4HL2IhJHI8bb0P8rZAxAqpJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1791540753; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=pmrQXpO04s5w4pu8FCEZISoi2E1afKc88b+8vgV8Uxw=; b=mnB4AubAjAKK8pyjLIzu938mCBbtClcXYIn/g4TzleFc9t+DGg2E0M0P5Xy6WYpzqPKsyA QZeGdHGXMzXeGoVrgAT0KjMfuToMbYrMfSnXcIaYELnNqjIl2mcASQtNRl0Y6xiyrxwo/E fTjaQzRmlvM2cBC91Y6dcSlO+FIeln6KniNZa9m4oaDggQSJZffdqUig1EqoZn5H/CP8Pb uE8dTixMtcywjyj3bg3RO+2Nh4NP6+KZJy6KjuU0AaVw0MOg21W7xv4KzGey4Vpjnoy5ku i/qP/6u72P48K0TwAgJjln1SnrYf5MBionW6/9vJNqNB/U4nVjiNJdy4eo7q/Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4j1N2n4Vhnz1LWj for ; Fri, 09 Oct 2026 10:12:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 41d76 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 09 Oct 2026 10:12:33 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Cc: Sumit Saxena From: Sumit Saxena Subject: git: 9e69060da5ea - main - if_bnxt: add HW-based TX completion coalescing List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ssaxena X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 9e69060da5eaa4a596be96a4a84b6484661466aa Auto-Submitted: auto-generated Date: Fri, 09 Oct 2026 10:12:33 +0000 Message-Id: <6ac8be11.41d76.370d1dec@gitrepo.freebsd.org> The branch main has been updated by ssaxena: URL: https://cgit.FreeBSD.org/src/commit/?id=9e69060da5eaa4a596be96a4a84b6484661466aa commit 9e69060da5eaa4a596be96a4a84b6484661466aa Author: Sumit Saxena AuthorDate: 2026-08-03 09:06:21 +0000 Commit: Sumit Saxena CommitDate: 2026-10-09 08:24:42 +0000 if_bnxt: add HW-based TX completion coalescing When firmware advertises BNXT_FLAG_TX_COAL_CMPL, request coalesced completions via cmpl_coal_cnt on TX ring alloc and decode the resulting tx_cmpl_coal_t records using a free-running SQ consumer index per ring, instead of one completion per BD. Add the tx_hw_coal_cnt sysctl to tune the firmware coalescing count code. Reviewed by: gallatin MFC after: 2 weeks Sponsored by: Broadcom Inc. Differential Revision: https://reviews.freebsd.org/D58599 --- sys/dev/bnxt/bnxt_en/bnxt.h | 8 ++++++++ sys/dev/bnxt/bnxt_en/bnxt_hwrm.c | 4 ++++ sys/dev/bnxt/bnxt_en/bnxt_sysctl.c | 33 +++++++++++++++++++++++++++++++++ sys/dev/bnxt/bnxt_en/bnxt_txrx.c | 31 +++++++++++++++++++++++++++++++ sys/dev/bnxt/bnxt_en/if_bnxt.c | 1 + 5 files changed, 77 insertions(+) diff --git a/sys/dev/bnxt/bnxt_en/bnxt.h b/sys/dev/bnxt/bnxt_en/bnxt.h index 35f1b9e97a00..ac91661eda7f 100644 --- a/sys/dev/bnxt/bnxt_en/bnxt.h +++ b/sys/dev/bnxt/bnxt_en/bnxt.h @@ -615,6 +615,7 @@ struct bnxt_ring { u32 db_key32; }; uint32_t db_ring_mask; + uint32_t free_flow_cons; /* HW TX-completion-coalescing SQ consumer */ uint32_t db_epoch_mask; uint8_t db_epoch_shift; @@ -1321,6 +1322,7 @@ struct bnxt_softc { #define BNXT_FW_CAP_CFA_NTUPLE_RX_EXT_IP_PROTO BIT_ULL(47) #define BNXT_FW_CAP_ENABLE_RDMA_SRIOV BIT_ULL(48) #define BNXT_FW_CAP_RSS_TCAM BIT_ULL(49) + #define BNXT_FLAG_TX_COAL_CMPL BIT_ULL(51) #define BNXT_FW_CAP_SW_MAX_RESOURCE_LIMITS BIT_ULL(61) #define BNXT_SW_RES_LMT(bp) ((bp)->fw_cap & BNXT_FW_CAP_SW_MAX_RESOURCE_LIMITS) @@ -1431,6 +1433,12 @@ struct bnxt_softc { #define BNXT_CAGR_TICK_RES_MAX 0x4 #define BNXT_CAGR_TICK_RES_OFFSET 16 uint32_t cagr_tick_res; + /* Firmware coalesces completions autonomously based on cmpl_coal_cnt + * (only valid when BNXT_FLAG_TX_COAL_CMPL is set); raw codes map to + * counts 4,8,12,16,24,32,48,64,96,128,192,256,320,384,MAX. */ +#define BNXT_TX_HW_COAL_CNT_DEFAULT \ + HWRM_RING_ALLOC_INPUT_CMPL_COAL_CNT_COAL_64 /* code 8 = up to 64 pkts */ + uint8_t tx_hw_coal_cnt; }; struct bnxt_filter_info { diff --git a/sys/dev/bnxt/bnxt_en/bnxt_hwrm.c b/sys/dev/bnxt/bnxt_en/bnxt_hwrm.c index 11d40a1a528a..6d5785a6fd5e 100644 --- a/sys/dev/bnxt/bnxt_en/bnxt_hwrm.c +++ b/sys/dev/bnxt/bnxt_en/bnxt_hwrm.c @@ -1278,6 +1278,8 @@ bnxt_hwrm_func_qcaps(struct bnxt_softc *softc) if (BNXT_PF(softc) && (flags_ext & HWRM_FUNC_QCAPS_OUTPUT_FLAGS_EXT_VF_CFG_ASYNC_FOR_PF_SUPPORTED)) softc->fw_cap |= BNXT_FW_CAP_VF_CFG_FOR_PF; + if (flags_ext & HWRM_FUNC_QCAPS_OUTPUT_FLAGS_EXT_TX_COAL_CMPL_CAP) + softc->fw_cap |= BNXT_FLAG_TX_COAL_CMPL; flags_ext2 = htole32(resp->flags_ext2); if (flags_ext2 & HWRM_FUNC_QCAPS_OUTPUT_FLAGS_EXT2_RX_ALL_PKTS_TIMESTAMPS_SUPPORTED) @@ -1984,6 +1986,8 @@ bnxt_hwrm_ring_alloc(struct bnxt_softc *softc, uint8_t type, req.stat_ctx_id = htole32(cp_ring->stats_ctx_id); req.enables |= htole32( HWRM_RING_ALLOC_INPUT_ENABLES_STAT_CTX_ID_VALID); + if (softc->fw_cap & BNXT_FLAG_TX_COAL_CMPL) + req.cmpl_coal_cnt = softc->tx_hw_coal_cnt; break; case HWRM_RING_ALLOC_INPUT_RING_TYPE_RX: if (!BNXT_CHIP_P5_PLUS(softc)) diff --git a/sys/dev/bnxt/bnxt_en/bnxt_sysctl.c b/sys/dev/bnxt/bnxt_en/bnxt_sysctl.c index c8a0f1962a13..729d0aa1f5c7 100644 --- a/sys/dev/bnxt/bnxt_en/bnxt_sysctl.c +++ b/sys/dev/bnxt/bnxt_en/bnxt_sysctl.c @@ -1580,6 +1580,30 @@ bnxt_set_cagr_tick_res(SYSCTL_HANDLER_ARGS) { return rc; } +static int +bnxt_set_tx_hw_coal_cnt(SYSCTL_HANDLER_ARGS) +{ + struct bnxt_softc *softc = arg1; + uint32_t val; + int rc; + + if (softc == NULL) + return EBUSY; + + val = softc->tx_hw_coal_cnt; + + rc = sysctl_handle_int(oidp, (int *)&val, 0, req); + if (rc || !req->newptr) + return rc; + + if (val > HWRM_RING_ALLOC_INPUT_CMPL_COAL_CNT_LAST) + return EINVAL; + + softc->tx_hw_coal_cnt = (uint8_t)val; + + return 0; +} + static void simulate_reset(struct bnxt_softc *bp, char *fwcli_string) { @@ -1675,6 +1699,15 @@ bnxt_create_config_sysctls_pre(struct bnxt_softc *softc) SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "cagr_tick_res", CTLTYPE_INT | CTLFLAG_RDTUN | CTLFLAG_MPSAFE, softc, 0, bnxt_set_cagr_tick_res, "I", "CAGR tick resolution"); + + softc->tx_hw_coal_cnt = BNXT_TX_HW_COAL_CNT_DEFAULT; + SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "tx_hw_coal_cnt", + CTLTYPE_INT | CTLFLAG_RDTUN | CTLFLAG_MPSAFE, softc, 0, + bnxt_set_tx_hw_coal_cnt, "I", + "HW-driven TX coalesced-completion packet count code for ring_alloc " + "(0=off 1=4 2=8 3=12 4=16 5=24 6=32 7=48 8=64[default] " + "9=96 10=128 11=192 12=256 13=320 14=384 15=MAX; " + "requires BNXT_FLAG_TX_COAL_CMPL firmware capability)"); SYSCTL_ADD_U32(ctx, children, OID_AUTO, "flags", CTLFLAG_RD, &softc->flags, 0, "flags"); SYSCTL_ADD_U64(ctx, children, OID_AUTO, "fw_cap", CTLFLAG_RD, diff --git a/sys/dev/bnxt/bnxt_en/bnxt_txrx.c b/sys/dev/bnxt/bnxt_en/bnxt_txrx.c index 8e95cca12a65..40b02744c6c1 100644 --- a/sys/dev/bnxt/bnxt_en/bnxt_txrx.c +++ b/sys/dev/bnxt/bnxt_en/bnxt_txrx.c @@ -225,6 +225,7 @@ bnxt_isc_txd_credits_update(void *sc, uint16_t txqid, bool clear) struct bnxt_softc *softc = (struct bnxt_softc *)sc; struct bnxt_cp_ring *cpr = &softc->tx_cp_rings[txqid]; struct tx_cmpl *cmpl = (struct tx_cmpl *)cpr->ring.vaddr; + struct bnxt_ring *txr; int avail = 0; uint32_t cons = cpr->cons; uint32_t raw_cons = cpr->raw_cons; @@ -266,6 +267,36 @@ bnxt_isc_txd_credits_update(void *sc, uint16_t txqid, bool clear) if (!clear) goto done; break; + case TX_CMPL_COAL_TYPE_TX_L2_COAL: + if (softc->fw_cap & BNXT_FLAG_TX_COAL_CMPL) { + tx_cmpl_coal_t *cmpl_coal = (tx_cmpl_coal_t *)cmpl; + uint32_t sq_cons; + + err = (le16toh(cmpl_coal[cons].errors_v) & + TX_CMPL_COAL_ERRORS_BUFFER_ERROR_MASK) >> + TX_CMPL_COAL_ERRORS_BUFFER_ERROR_SFT; + if (err) + device_printf(softc->dev, + "TX completion error %u\n", err); + + sq_cons = le32toh(cmpl_coal[cons].sq_cons_idx) & + TX_CMPL_COAL_SQ_CONS_IDX_MASK; + + txr = &softc->tx_rings[txqid]; + avail += (sq_cons - txr->free_flow_cons) & + txr->db_ring_mask; + + /* + * Only the consuming (clear) pass advances free_flow_cons; + * peek calls must not claim unreclaimed SQ entries. Gating + * on clear also means only gtaskq writes it, so no atomics. + */ + if (!clear) + goto done; + txr->free_flow_cons = sq_cons; + break; + } + /* FALLTHROUGH */ default: if (type & 1) { NEXT_CP_CONS_V(&cpr->ring, cons, v_bit); diff --git a/sys/dev/bnxt/bnxt_en/if_bnxt.c b/sys/dev/bnxt/bnxt_en/if_bnxt.c index 188311b2d7d4..512afb450494 100644 --- a/sys/dev/bnxt/bnxt_en/if_bnxt.c +++ b/sys/dev/bnxt/bnxt_en/if_bnxt.c @@ -3512,6 +3512,7 @@ skip_def_cp_ring: HWRM_RING_ALLOC_INPUT_RING_TYPE_TX); if (rc) goto fail; + softc->tx_rings[i].free_flow_cons = 0; softc->db_ops.bnxt_db_tx(&softc->tx_rings[i], 0); }