From nobody Thu Oct 29 06:02:58 2026 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4jFYYb1ZV9z6xrWD for ; Thu, 29 Oct 2026 06:02:59 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4jFYYZ2XJTz4ryP for ; Thu, 29 Oct 2026 06:02:58 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1793253778; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=2HnZoZZHA1xDhCWiT0hoxEAXgP5jqNRs+3spXAVuetQ=; b=LoTG/PaLGzTp8b/qu+FH5Og54clok8FwVzeSzqW29XNrU0Sp9SrXZblfVxpVQIozDwKCtn YbActAgkZo9phVP3GyBfUZaRZqeMPJ9jAgOlnERoK/ZcW4Kae7R7NWfK+m4dpR1ZJDYzAN Heg4j2YfTjpuTxuJ/PtInRJqNO8wP9BXnKq+nIRSxFhYQNOMEv0eUoBGbtegnZd75NFoqN MiKlyDpDvkolm4QCb3VYbQKi5Irq5eyOjSTU++MUBeSZ4YzcGKSfFSX49q6fA4jdMJ2eyQ j56po5mtp3+wlb8dcCNHSDBUDNlI4hlLvYjcSTc1HcN78iSd/Fp508FSKWI8+A== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1793253778; b=c7G2UrDGprr9uZzHJKcpqALfnJw6l7WcGMJ6bJlpwxXhREEs3vWnDQ3fkx/16EAVNt2EGV 6pnwZtCne6nwLV7u6cmSmz4Jp3ebX5sgUszpzv+b5Tih1H1IXOu0l42V/m+oP/OXxXB50Y nNm/4CvFUJPQKrOfO7Cf4/5AtJ795YcBhTkxO1OU+qeUvJ8gzalGRAnQZTiVgoMkdRTMpL 85jr8yIbQcq3JNr9R601BANtiOwiM4T7xfoASngOEN/9aer0jzM1TvGtWcCJIaw28kEO4s aZw3Bd0uY9pKTTYWQ9KWIff/6t+tuz7nBbSUWNPK8TikNay87oMMB+ABnBkgmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1793253778; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=2HnZoZZHA1xDhCWiT0hoxEAXgP5jqNRs+3spXAVuetQ=; b=KbFYmghpg9p9xhaR1jn2Skj54XMEI9AYokz6kVCLIt7Te3Hv5JCx39UNh+LsIeLwuNhjjw aBuzIEGvEsEzUGA5NQVKLjbdAkJq4lGcMyUzHlN7fXUZssTa30JKrzyh+VYBinxUNB+9DS nNhQ9ay2hUmj2LwF7HHFk57QfJ2E/Vkj3o56+8Ef86Z1u4+ifMhKcyuHN3NxBEucWhUDuL IncLmlszDN2VkI5iTrUTw8IZbdGICQrHF7JHi5gKXXV1KYjGJHjib/T3Vqf5WwYBgJABXU kYwJ6pM0fdJIBbwWhSGqv9vHAkmAqa0bB9JFK8hNqd63Af5BTELS2vVyCmk2jQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4jFYYZ1m9YzR9Q for ; Thu, 29 Oct 2026 06:02:58 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3a398 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Thu, 29 Oct 2026 06:02:58 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org Cc: Andrew Gallatin From: Kevin Bowling Subject: git: bb3e69035ad0 - stable/15 - iflib: Implement tx desc reclaim threshold List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org List-Id: Commits to the stable branches of the FreeBSD src repository List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: bb3e69035ad0266e6c0fcb9d23a9ad655c45faa0 Auto-Submitted: auto-generated Date: Thu, 29 Oct 2026 06:02:58 +0000 Message-Id: <6ae2e192.3a398.78c7e629@gitrepo.freebsd.org> The branch stable/15 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=bb3e69035ad0266e6c0fcb9d23a9ad655c45faa0 commit bb3e69035ad0266e6c0fcb9d23a9ad655c45faa0 Author: Andrew Gallatin AuthorDate: 2025-10-01 22:43:05 +0000 Commit: Kevin Bowling CommitDate: 2026-10-29 05:28:00 +0000 iflib: Implement tx desc reclaim threshold On some iflib drivers, the txd reclaim routine can be fairly expensive at high packet rates. Iflib was designed with the intent of only reclaiming tx descriptors above a configurable threshold, but this logic was left unimplemented. This change: - implements 2 new knobs, iflib.tx_reclaim_thresh and iflib.tx_reclaim_ticks. - moves tx reclaim thresh from the if_shared_ctx and into the iflib_ctx as drivers don't need to see it, and it needs to be changed, so it can't be const - tx_reclaim_thresh and ticks are replicated into the txq to improve cache locality of data accessed in the hot path - ticks is used rather than more expensive timekeeping mechanism so as to keep things simple and cheap This change substantially improves packet rates on bnxt. It has been tested on bxnt and ixl Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D52561 Reviewed by: markj (initial version) (cherry picked from commit 213170eb956fa7eec93c2ab4159d6ed68e8e1b1d) --- sys/net/iflib.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++------ sys/net/iflib.h | 2 +- 2 files changed, 101 insertions(+), 13 deletions(-) diff --git a/sys/net/iflib.c b/sys/net/iflib.c index a644378ec10c..01b41ce37a7e 100644 --- a/sys/net/iflib.c +++ b/sys/net/iflib.c @@ -242,6 +242,8 @@ struct iflib_ctx { bool ifc_cpus_are_physical_cores; bool ifc_core_offset_ref; bool ifc_sysctl_simple_tx; + uint16_t ifc_sysctl_tx_reclaim_thresh; + uint16_t ifc_sysctl_tx_reclaim_ticks; qidx_t ifc_sysctl_ntxds[8]; qidx_t ifc_sysctl_nrxds[8]; @@ -385,7 +387,9 @@ struct iflib_txq { uint16_t ift_npending; uint16_t ift_db_pending; uint16_t ift_rs_pending; - /* implicit pad */ + uint32_t ift_last_reclaim; + uint16_t ift_reclaim_thresh; + uint16_t ift_reclaim_ticks; uint8_t ift_txd_size[8]; uint64_t ift_processed; uint64_t ift_cleaned; @@ -744,7 +748,7 @@ static void iflib_free_intr_mem(if_ctx_t ctx); #ifndef __NO_STRICT_ALIGNMENT static struct mbuf *iflib_fixup_rx(struct mbuf *m); #endif -static __inline int iflib_completed_tx_reclaim(iflib_txq_t txq, int thresh); +static __inline int iflib_completed_tx_reclaim(iflib_txq_t txq); static SLIST_HEAD(cpu_offset_list, cpu_offset) cpu_offsets = SLIST_HEAD_INITIALIZER(cpu_offsets); @@ -3167,8 +3171,6 @@ txq_max_rs_deferred(iflib_txq_t txq) #define QIDX(ctx, m) ((((m)->m_pkthdr.flowid & ctx->ifc_softc_ctx.isc_rss_table_mask) % NTXQSETS(ctx)) + FIRST_QSET(ctx)) #define DESC_RECLAIMABLE(q) ((int)((q)->ift_processed - (q)->ift_cleaned - (q)->ift_ctx->ifc_softc_ctx.isc_tx_nsegments)) -/* XXX we should be setting this to something other than zero */ -#define RECLAIM_THRESH(ctx) ((ctx)->ifc_sctx->isc_tx_reclaim_thresh) #define MAX_TX_DESC(ctx) MAX((ctx)->ifc_softc_ctx.isc_tx_tso_segments_max, \ (ctx)->ifc_softc_ctx.isc_tx_nsegments) @@ -3736,7 +3738,7 @@ defrag: * cxgb */ if (__predict_false(nsegs > TXQ_AVAIL(txq))) { - (void)iflib_completed_tx_reclaim(txq, RECLAIM_THRESH(ctx)); + (void)iflib_completed_tx_reclaim(txq); if (__predict_false(nsegs > TXQ_AVAIL(txq))) { txq->ift_no_desc_avail++; bus_dmamap_unload(buf_tag, map); @@ -3878,14 +3880,21 @@ iflib_tx_desc_free(iflib_txq_t txq, int n) } static __inline int -iflib_completed_tx_reclaim(iflib_txq_t txq, int thresh) +iflib_completed_tx_reclaim(iflib_txq_t txq) { - int reclaim; + int reclaim, thresh; + uint32_t now; if_ctx_t ctx = txq->ift_ctx; + thresh = txq->ift_reclaim_thresh; KASSERT(thresh >= 0, ("invalid threshold to reclaim")); MPASS(thresh /*+ MAX_TX_DESC(txq->ift_ctx) */ < txq->ift_size); + now = ticks; + if (now <= (txq->ift_last_reclaim + txq->ift_reclaim_ticks) && + txq->ift_in_use < thresh) + return (0); + txq->ift_last_reclaim = now; /* * Need a rate-limiting check so that this isn't called every time */ @@ -3966,7 +3975,7 @@ iflib_txq_drain(struct ifmp_ring *r, uint32_t cidx, uint32_t pidx) DBG_COUNTER_INC(txq_drain_notready); return (0); } - reclaimed = iflib_completed_tx_reclaim(txq, RECLAIM_THRESH(ctx)); + reclaimed = iflib_completed_tx_reclaim(txq); rang = iflib_txd_db_check(txq, reclaimed && txq->ift_db_pending); avail = IDXDIFF(pidx, cidx, r->size); @@ -4045,7 +4054,7 @@ iflib_txq_drain(struct ifmp_ring *r, uint32_t cidx, uint32_t pidx) } /* deliberate use of bitwise or to avoid gratuitous short-circuit */ - ring = rang ? false : (iflib_min_tx_latency | err); + ring = rang ? false : (iflib_min_tx_latency | err | (!!txq->ift_reclaim_thresh)); iflib_txd_db_check(txq, ring); if_inc_counter(ifp, IFCOUNTER_OBYTES, bytes_sent); if_inc_counter(ifp, IFCOUNTER_OPACKETS, pkt_sent); @@ -4125,7 +4134,7 @@ _task_fn_tx(void *context) #endif if (ctx->ifc_sysctl_simple_tx) { mtx_lock(&txq->ift_mtx); - (void)iflib_completed_tx_reclaim(txq, RECLAIM_THRESH(ctx)); + (void)iflib_completed_tx_reclaim(txq); mtx_unlock(&txq->ift_mtx); goto skip_ifmp; } @@ -6248,6 +6257,7 @@ iflib_queues_alloc(if_ctx_t ctx) device_printf(dev, "Unable to allocate buf_ring\n"); goto err_tx_desc; } + txq->ift_reclaim_thresh = ctx->ifc_sysctl_tx_reclaim_thresh; } for (i = 0; i < nrxqsets; i++, rxq++) { @@ -7138,6 +7148,74 @@ mp_ndesc_handler(SYSCTL_HANDLER_ARGS) return (rc); } +static int +iflib_handle_tx_reclaim_thresh(SYSCTL_HANDLER_ARGS) +{ + if_ctx_t ctx = (void *)arg1; + iflib_txq_t txq; + int i, err; + int thresh; + + thresh = ctx->ifc_sysctl_tx_reclaim_thresh; + err = sysctl_handle_int(oidp, &thresh, arg2, req); + if (err != 0) { + return err; + } + + if (thresh == ctx->ifc_sysctl_tx_reclaim_thresh) + return 0; + + if (thresh > ctx->ifc_softc_ctx.isc_ntxd[0] / 2) { + device_printf(ctx->ifc_dev, "TX Reclaim thresh must be <= %d\n", + ctx->ifc_softc_ctx.isc_ntxd[0] / 2); + return (EINVAL); + } + + ctx->ifc_sysctl_tx_reclaim_thresh = thresh; + if (ctx->ifc_txqs == NULL) + return (err); + + txq = &ctx->ifc_txqs[0]; + for (i = 0; i < NTXQSETS(ctx); i++, txq++) { + txq->ift_reclaim_thresh = thresh; + } + return (err); +} + +static int +iflib_handle_tx_reclaim_ticks(SYSCTL_HANDLER_ARGS) +{ + if_ctx_t ctx = (void *)arg1; + iflib_txq_t txq; + int i, err; + int ticks; + + ticks = ctx->ifc_sysctl_tx_reclaim_ticks; + err = sysctl_handle_int(oidp, &ticks, arg2, req); + if (err != 0) { + return err; + } + + if (ticks == ctx->ifc_sysctl_tx_reclaim_ticks) + return 0; + + if (ticks > hz) { + device_printf(ctx->ifc_dev, + "TX Reclaim ticks must be <= hz (%d)\n", hz); + return (EINVAL); + } + + ctx->ifc_sysctl_tx_reclaim_ticks = ticks; + if (ctx->ifc_txqs == NULL) + return (err); + + txq = &ctx->ifc_txqs[0]; + for (i = 0; i < NTXQSETS(ctx); i++, txq++) { + txq->ift_reclaim_ticks = ticks; + } + return (err); +} + #define NAME_BUFLEN 32 static void iflib_add_device_sysctl_pre(if_ctx_t ctx) @@ -7223,6 +7301,16 @@ iflib_add_device_sysctl_post(if_ctx_t ctx) node = ctx->ifc_sysctl_node; child = SYSCTL_CHILDREN(node); + SYSCTL_ADD_PROC(ctx_list, child, OID_AUTO, "tx_reclaim_thresh", + CTLTYPE_INT | CTLFLAG_RWTUN, ctx, + 0, iflib_handle_tx_reclaim_thresh, "I", + "Number of TX descs outstanding before reclaim is called"); + + SYSCTL_ADD_PROC(ctx_list, child, OID_AUTO, "tx_reclaim_ticks", + CTLTYPE_INT | CTLFLAG_RWTUN, ctx, + 0, iflib_handle_tx_reclaim_ticks, "I", + "Number of ticks before a TX reclaim is forced"); + if (scctx->isc_ntxqsets > 100) qfmt = "txq%03d"; else if (scctx->isc_ntxqsets > 10) @@ -7492,7 +7580,7 @@ iflib_debugnet_poll(if_t ifp, int count) return (EBUSY); txq = &ctx->ifc_txqs[0]; - (void)iflib_completed_tx_reclaim(txq, RECLAIM_THRESH(ctx)); + (void)iflib_completed_tx_reclaim(txq); NET_EPOCH_ENTER(et); for (i = 0; i < scctx->isc_nrxqsets; i++) @@ -7542,7 +7630,7 @@ iflib_simple_transmit(if_t ifp, struct mbuf *m) else if_inc_counter(ifp, IFCOUNTER_OERRORS, 1); } - (void)iflib_completed_tx_reclaim(txq, RECLAIM_THRESH(ctx)); + (void)iflib_completed_tx_reclaim(txq); mtx_unlock(&txq->ift_mtx); if_inc_counter(ifp, IFCOUNTER_OBYTES, bytes_sent); if_inc_counter(ifp, IFCOUNTER_OPACKETS, pkt_sent); diff --git a/sys/net/iflib.h b/sys/net/iflib.h index 844ee1bb9594..0b92c18858cc 100644 --- a/sys/net/iflib.h +++ b/sys/net/iflib.h @@ -273,7 +273,7 @@ struct if_shared_ctx { int isc_ntxqs; /* # of tx queues per tx qset - usually 1 */ int isc_nrxqs; /* # of rx queues per rx qset - intel 1, chelsio 2, broadcom 3 */ int __spare0__; - int isc_tx_reclaim_thresh; + int __spare1__; int isc_flags; };