From nobody Wed Sep 02 17:40:00 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hZqkF5LkKz6qQ9V for ; Wed, 02 Sep 2026 17:40:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hZqkF4kWkz46dB for ; Wed, 02 Sep 2026 17:40:05 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1788370805; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=26/hRAYp7/xEbh7qfVk3VILBYowbXT0zUqA2/y2EVwI=; b=vmnZWng5BBTGp1orIQ8HigoprInHeOKI5th2fa5MFpbssHx7aqHcCnCbrU+zbrnzJ2CYrO Lg8CauNYyjwLPUgVOft7bEqNUN63tMypeD5q/hghq0lGd6/NaDFaAKZeaNJcN8RxsB4AbD 4PX1qsoWkDWkd+Klpy9uahpa9/q30Sp07vyeAUzG4WZJZm0nea2pJfVQoDPw9iDp0hOFjS Wfr3BRiEeT2mNKGhZSkKOPfnf0WyVfmzBPtPNtxgaYv9vGGe/RIsuQuYzRBKOXwaBJTM8a RZBXzvfcEJlrkuWa4ba3DYuqa2GAMq6UU+4EqdS/upEM8wUjaK9f02852rr8aw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1788370805; a=rsa-sha256; cv=none; b=CGNcRjShhFBMPKhoKxb/E3C6yd7ha2IFnzG6TTnLhZHKnL7cQYJjTl/LK44Z65F8h+NjeX Mcv7NR8bV6W64iS14Wyjvr1AotfosNe7mpY+mX9VbJ+wqhV3pogRSg69wRPF2pxH+WHd6h W9NBWQT821K619Tlw6dDebrdQIeFFNKR94HESIBszZaUu+6myMmjJBA9xbyk8fA646XfrN byi++Zzk9yCwcwIVcIuqUb0lII51WDElw5HvLy1IaLmuRXkBvUxV5lc4k6B2g4Vof5tMjU 1qiTsxbca/aQCThBKagpUvrYqiYcgsF22pbGKR3/Q8wajFXgUR3y8EoEXLazvA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1788370805; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=26/hRAYp7/xEbh7qfVk3VILBYowbXT0zUqA2/y2EVwI=; b=UrmikerNtLdT/IiJpruq53x5f5mgQtcxXNKJYTYeq59FBGuDFI13PfKcjFI2CYdP1mQfGY zAlobxCdFz8jGO4ES7VkYGK95V50RjkNgB9Cbv8tdWEfG4Cg6owNsLhGxhfvUDhy3D4vW4 Mr5AWwremlaeZJYYhYf4myfuiTAgas0aA+ySSsj9BCKUVLHueTklfl7okshRXaTQox5Wnh ymTxnbbmb2keSl8q9CMyHRyUpZaF6/mSairTeeu3z8ancXHStOTR8A8z/x3zjl/Yq7kWV2 JEpb7MWjTPhvSpclYPb3j3+6eeqNJyN0yGG25VEjVx1N4Gl4dEuLGVSJaIdv7w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hZqkF3fYkztHQ for ; Wed, 02 Sep 2026 17:40:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3cce2 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Wed, 02 Sep 2026 17:40:00 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Kevin Bowling Subject: git: a550e8280b98 - main - iflib: Require sustained demand for TX watchdog List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: a550e8280b9881c8c207b842f51065f155b84951 Auto-Submitted: auto-generated Date: Wed, 02 Sep 2026 17:40:00 +0000 Message-Id: <6a985f70.3cce2.2b47d91b@gitrepo.freebsd.org> The branch main has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=a550e8280b9881c8c207b842f51065f155b84951 commit a550e8280b9881c8c207b842f51065f155b84951 Author: Kevin Bowling AuthorDate: 2026-08-27 02:00:34 +0000 Commit: Kevin Bowling CommitDate: 2026-09-02 17:36:20 +0000 iflib: Require sustained demand for TX watchdog The restored watchdog arms when the outstanding descriptor count grows, but then continues counting based only on the queue remaining frozen. A single growth sample can therefore leave a quiet, nearly empty queue armed until the watchdog resets the interface. Lockless sampling of the queue counters can also manufacture the initial growth sample. This matches watchdog reports from I354 queues with 979 or 980 of 1022 usable descriptors still available. Neither queue was under transmit backpressure when the reset flapped its link. Keep the watchdog armed only while the outstanding count continues to grow, the software ring is stalled, or the hardware ring is at iflib's backpressure threshold. The last condition preserves hang detection with simple-TX, which bypasses the software ring. A busy hang still reaches the verdict while a frozen but quiet tail disarms. Retain the final driver completion peek so a missed completion interrupt schedules the queue task instead of resetting it. Validated on an 82580 with one and four queue sets in the default mp_ring and simple-TX modes. Sustained traffic and repeated burst/idle cycles produced no false resets. Sixteen-flow runs exercised all four queues in both modes. Clearing TCTL.EN under load in each configuration filled the rings; the reset counter advanced once per injection, reset restored TCTL and the link, and traffic recovered. Tested by: glebius Reviewed by: iflib (gallatin), manpages (ziaee) Fixes: 69c3e0de01c1 ("iflib: restore TX watchdog functionality") MFC after: 6 days (after 69c3e0de01c1) Sponsored by: BBOX.io --- share/man/man4/iflib.4 | 14 +++++++----- sys/net/iflib.c | 60 ++++++++++++++++++++++++++++---------------------- 2 files changed, 43 insertions(+), 31 deletions(-) diff --git a/share/man/man4/iflib.4 b/share/man/man4/iflib.4 index 4cf08602fd04..fb8bb37413c7 100644 --- a/share/man/man4/iflib.4 +++ b/share/man/man4/iflib.4 @@ -1,4 +1,4 @@ -.Dd August 8, 2026 +.Dd August 26, 2026 .Dt IFLIB 4 .Os .Sh NAME @@ -131,10 +131,14 @@ Setting this to a non-zero value will disable the use of this feature. .It Va net.iflib.tx_watchdog_periods Number of consecutive .Va net.iflib.timer_default -intervals for which a transmit queue must stay frozen, -that is, hold descriptors the hardware has not reported as completed -while none are reclaimed, -before iflib asks the hardware whether it has completions pending. +intervals for which a transmit queue must remain frozen under demand before +.Nm +asks the hardware whether it has completions pending. +A queue is frozen when it holds descriptors the hardware has not reported as +completed while none are reclaimed. +Demand persists while the outstanding descriptor count grows, the software +queue is stalled, or the hardware descriptor ring is at its backpressure +threshold. If it reports none, the interface is reset. Setting this to zero disables the check. .El diff --git a/sys/net/iflib.c b/sys/net/iflib.c index 37f1218bbb17..34d53d648a7c 100644 --- a/sys/net/iflib.c +++ b/sys/net/iflib.c @@ -481,6 +481,9 @@ get_inuse(int size, qidx_t cidx, qidx_t pidx, uint8_t gen) #define TXQ_AVAIL(txq) ((txq->ift_size - txq->ift_pad) -\ get_inuse(txq->ift_size, txq->ift_cidx, txq->ift_pidx, txq->ift_gen)) +#define MAX_TX_DESC(ctx) MAX((ctx)->ifc_softc_ctx.isc_tx_tso_segments_max, \ + (ctx)->ifc_softc_ctx.isc_tx_nsegments) + #define IDXDIFF(head, tail, wrap) \ ((head) >= (tail) ? (head) - (tail) : (wrap) - (tail) + (head)) @@ -596,19 +599,19 @@ static int iflib_timer_default = 1000; SYSCTL_INT(_net_iflib, OID_AUTO, timer_default, CTLFLAG_RW, &iflib_timer_default, 0, "number of ticks between iflib_timer calls"); /* - * Consecutive timer periods a TX queue must stay frozen - see - * iflib_timer(), which defines that state - before the hardware is - * asked whether it has completions pending. Four periods is roughly - * two seconds with the default timer interval: a healthy queue on - * hardware that coalesces completion reports (e.g. 8254x, - * TXDCTL.WTHRESH) stays frozen for at most two (measured on 82541PI), - * a wedged one until it is reset. + * Consecutive timer periods a TX queue must stay frozen while demand + * persists - see iflib_timer(), which defines those states - before the + * hardware is asked whether it has completions pending. Four periods is + * roughly two seconds with the default timer interval: a healthy queue on + * hardware that coalesces completion reports (e.g. 8254x, TXDCTL.WTHRESH) + * stays frozen for at most two (measured on 82541PI), a wedged one until it + * is reset. */ static int iflib_tx_watchdog_periods = 4; SYSCTL_INT(_net_iflib, OID_AUTO, tx_watchdog_periods, CTLFLAG_RWTUN, &iflib_tx_watchdog_periods, 0, - "consecutive frozen timer periods before a TX queue is checked for " - "a hang (0 disables the check)"); + "consecutive frozen timer periods under demand before a TX queue is " + "checked for a hang (0 disables the check)"); #if IFLIB_DEBUG_COUNTERS @@ -2447,8 +2450,8 @@ iflib_timer(void *arg) * delays the verdict by one timer period. */ if (this_tick - txq->ift_last_timer_tick >= iflib_timer_default) { - qidx_t outstanding; - bool frozen; + qidx_t in_use, outstanding; + bool demand, frozen; txq->ift_last_timer_tick = this_tick; IFDI_TIMER(ctx, txq->ift_id); @@ -2462,7 +2465,8 @@ iflib_timer(void *arg) * reported and must not count (ift_rs_pending * over-counts it by one per packet). */ - outstanding = txq->ift_in_use - + in_use = txq->ift_in_use; + outstanding = in_use - (qidx_t)(txq->ift_processed - txq->ift_cleaned); /* @@ -2472,12 +2476,17 @@ iflib_timer(void *arg) * up, with no pause frames and no pending doorbell * (the laggard check below rings it). * - * Being frozen is not a fault - the hardware may - * defer marking descriptors as completed - * indefinitely, and 8254x hardware does so for a - * quiet queue - therefore the check arms only when a - * frozen queue also takes on new work, and acts only - * once it has stayed frozen for + * Being frozen is not a fault - the hardware may defer + * marking descriptors as completed indefinitely, and + * 8254x hardware does so for a quiet queue. Continue + * arming only while demand persists: the outstanding + * count grows, the software ring is stalled, or the + * hardware ring has reached iflib's backpressure + * threshold. The last condition covers simple-TX, which + * does not use the software ring. This also prevents one + * mixed lockless counter sample from arming a quiet queue + * until the verdict. Act only once it has stayed frozen + * under demand for * net.iflib.tx_watchdog_periods consecutive periods. */ frozen = outstanding > txq->ift_rs_pending && @@ -2485,10 +2494,12 @@ iflib_timer(void *arg) txq->ift_db_pending == 0 && sctx->isc_pause_frames == 0 && ctx->ifc_link_state == LINK_STATE_UP; - if (!frozen) + demand = outstanding > txq->ift_outstanding_prev || + ifmp_ring_is_stalled(txq->ift_br) || + in_use + MAX_TX_DESC(ctx) >= txq->ift_size - txq->ift_pad; + if (!frozen || !demand) txq->ift_wdog_armed = 0; - else if (txq->ift_wdog_armed > 0 || - outstanding > txq->ift_outstanding_prev) { + else { if (txq->ift_wdog_armed < UINT16_MAX) txq->ift_wdog_armed++; } @@ -2497,8 +2508,8 @@ iflib_timer(void *arg) * Frozen long enough: ask the hardware. Completions * ready but unharvested for this long mean the * completion interrupt went missing - kick the - * queue's task. Nothing ready, although the queue - * kept taking on work, means it is hung. + * queue's task. Nothing ready while demand persisted + * means it is hung. */ if (iflib_tx_watchdog_periods > 0 && txq->ift_wdog_armed >= iflib_tx_watchdog_periods) { @@ -3192,9 +3203,6 @@ txq_max_rs_deferred(iflib_txq_t txq) #define QIDX(ctx, m) ((((m)->m_pkthdr.flowid & ctx->ifc_softc_ctx.isc_rss_table_mask) % NTXQSETS(ctx)) + FIRST_QSET(ctx)) #define DESC_RECLAIMABLE(q) ((int)((q)->ift_processed - (q)->ift_cleaned - (q)->ift_ctx->ifc_softc_ctx.isc_tx_nsegments)) -#define MAX_TX_DESC(ctx) MAX((ctx)->ifc_softc_ctx.isc_tx_tso_segments_max, \ - (ctx)->ifc_softc_ctx.isc_tx_nsegments) - static inline bool iflib_txd_db_check(iflib_txq_t txq, int ring) {