From nobody Fri Oct 09 10:12:53 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4j1N3B2Wctz6vRVK for ; Fri, 09 Oct 2026 10:12:54 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4j1N396qmKz4Y7h for ; Fri, 09 Oct 2026 10:12:53 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1791540774; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ATpkhTz2hWF7SpWcwffv70Yh8P+RSzpEf46YtBqny+0=; b=lANBbzYRnOOpDGZ7t9jNRUFr5ylfLm+H+u8FymqcTl6BIkT9SgRFJObd0BF0ShY1bthryq lFQAUyPknk58/r1HymfONI3NKUcA7Pu+gwaUtGI+5BIF88VNlmOVvTjn3801VZw0rfhhkg pZrwBjHCSEm5hdGI3HV55VMXbELZkha4c+8e1wu6TrNi1iY0KOiVKPV+0OtslVo/iPsyf7 eoS7GeHHt/Fx0Gz8wz1XIJ0a/Ur02dTCdYg2G7xJ9ejdS087N2Rf3mqN3xyWCzAKu6PYHC ThXWWQ0fFFG94A3ti9qwGwKjTei0rHA476hYLaIjoZYI3qWCAqMlRy4yB4hZng== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1791540774; b=IGYmgBKoz6IIVuzhug7kLSXXpUuJtPa6idtO6THn7YFA/E545kKYfj/DENHgHRc482wn3q kHSD/wfB8C6wgStPizuuQy6zMzhS1kMT8JxrWInXFE7rjbQa7u5MZ+6XDs2oXAcWkUlSdT 7JKd56Vl7irFiBnzY9Rm/9LUkQGTkOQyKucOHGcXVnmTI95ht6tCjKQ7wVmtX3lMpz6CYg 7JsUIv11Y0o2GUpr8PPu4ci1Gnsl3WViGK2Lnxqx/wo7wUEA3lbOu1hhp8mxnSRiRFtma9 d6Z82RyDRKqXwGFk+5VvZW+dGwR74JyGRkrn375iIcSeqcVMXNiH8Dr9FJBlnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1791540774; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ATpkhTz2hWF7SpWcwffv70Yh8P+RSzpEf46YtBqny+0=; b=vQA8TNBeZxiQhSJDm+JDEnaFBetF7qqywJ8AECvdt4cmd2YUqoL2z/MOLzt5/XT1/ci1+e 3OPHW49fEuvB2BkuzQ8FfxGA9o03dAW6cL+D4dvJ0WV3bO/EFlQ/hlvqMGAAVxBLLHIJrY Tx4udGVBL7A+XLH7S/Mr3+q7m8voc1+ekssLMvpq5wDLyG3k2L6xAXZKEWSTNSCEXptZ8b g/ee/THWVMei16gSWvaDNp4ZnzzX7Yk90pjIXvLiQU1mcwLvsfWeolqvrQFQbHCTsI7XYU EiKzmgCiULNC+qEEcXs8lzDYfgoO2XIVJGO7abIOreGCWrlVGXCgVmgyAFp4fA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4j1N393TQHz1LHx for ; Fri, 09 Oct 2026 10:12:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 42c20 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 09 Oct 2026 10:12:53 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Cc: Sumit Saxena From: Sumit Saxena Subject: git: dbde45975144 - main - if_bnxt: fix HWRM failures/timeouts after repeated FW resets List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ssaxena X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: dbde459751448ab9e98662660049513db783600c Auto-Submitted: auto-generated Date: Fri, 09 Oct 2026 10:12:53 +0000 Message-Id: <6ac8be25.42c20.2d12bed3@gitrepo.freebsd.org> The branch main has been updated by ssaxena: URL: https://cgit.FreeBSD.org/src/commit/?id=dbde459751448ab9e98662660049513db783600c commit dbde459751448ab9e98662660049513db783600c Author: Sumit Saxena AuthorDate: 2026-07-31 16:13:44 +0000 Commit: Sumit Saxena CommitDate: 2026-10-09 08:24:45 +0000 if_bnxt: fix HWRM failures/timeouts after repeated FW resets Consecutive firmware-initiated reset cycles produced HWRM failures/timeouts and traffic didn't come back. Fix the FW-reset recovery path: - bnxt_fw_reset_close() called bnxt_stop() and bnxt_hwrm_func_drv_unrgtr(), sending HWRM ring/VNIC/filter free commands to a firmware that's already mid-reset and unresponsive, producing the observed timeouts. Both are unnecessary since firmware comes back with fresh state anyway; drop them along with the now-redundant iflib_request_reset(). - bnxt_func_reset() now skips bnxt_hwrm_resource_free() entirely while BNXT_STATE_IN_FW_RESET is set, for the same reason. - bnxt_open() (used to reopen after a firmware reset) now issues bnxt_hwrm_func_reset() up front and drives reinit through iflib's own reset machinery (iflib_request_reset() plus a new bnxt_iflib_reset_sync() that waits for IFF_DRV_RUNNING/OACTIVE to flip 1->0->1) instead of calling bnxt_init()/bnxt_intr_enable() directly. - Route repeated PORT_QSTATS/PORT_QSTATS_EXT HWRM timeouts through bnxt_log_live() instead of device_printf(), so a stats poll timing out during a reset doesn't spam dmesg on every tick. Reviewed by: gallatin MFC after: 2 weeks Sponsored by: Broadcom Inc. Differential Revision: https://reviews.freebsd.org/D58617 --- sys/dev/bnxt/bnxt_en/bnxt.h | 1 + sys/dev/bnxt/bnxt_en/bnxt_hwrm.c | 15 ++++++--- sys/dev/bnxt/bnxt_en/if_bnxt.c | 67 ++++++++++++++++++++++++++++++++-------- 3 files changed, 66 insertions(+), 17 deletions(-) diff --git a/sys/dev/bnxt/bnxt_en/bnxt.h b/sys/dev/bnxt/bnxt_en/bnxt.h index 763748974458..37914dae9a1d 100644 --- a/sys/dev/bnxt/bnxt_en/bnxt.h +++ b/sys/dev/bnxt/bnxt_en/bnxt.h @@ -50,6 +50,7 @@ #include "bnxt_dcb.h" #include "bnxt_auxbus_compat.h" #include "bnxt_sriov.h" +#include "bnxt_log.h" #define DFLT_HWRM_CMD_TIMEOUT 500 diff --git a/sys/dev/bnxt/bnxt_en/bnxt_hwrm.c b/sys/dev/bnxt/bnxt_en/bnxt_hwrm.c index 19f2a8c80485..b9cd7174209c 100644 --- a/sys/dev/bnxt/bnxt_en/bnxt_hwrm.c +++ b/sys/dev/bnxt/bnxt_en/bnxt_hwrm.c @@ -256,10 +256,17 @@ _hwrm_send_message(struct bnxt_softc *softc, void *msg, uint32_t msg_len) DELAY(1000); } if (i >= softc->hwrm_cmd_timeo) { - device_printf(softc->dev, - "Timeout sending %s: (timeout: %u) seq: %d\n", - GET_HWRM_REQ_TYPE(req->req_type), softc->hwrm_cmd_timeo, - le16toh(req->seq_id)); + if (req->req_type == HWRM_PORT_QSTATS || + req->req_type == HWRM_PORT_QSTATS_EXT) + bnxt_log_live(softc, BNXT_LOGGER_L2, + "Timeout sending %s: (timeout: %u) seq: %d\n", + GET_HWRM_REQ_TYPE(req->req_type), softc->hwrm_cmd_timeo, + le16toh(req->seq_id)); + else + device_printf(softc->dev, + "Timeout sending %s: (timeout: %u) seq: %d\n", + GET_HWRM_REQ_TYPE(req->req_type), softc->hwrm_cmd_timeo, + le16toh(req->seq_id)); return ETIMEDOUT; } /* Last byte of resp contains the valid key */ diff --git a/sys/dev/bnxt/bnxt_en/if_bnxt.c b/sys/dev/bnxt/bnxt_en/if_bnxt.c index 2d996d80a0f5..b7dd72ffb177 100644 --- a/sys/dev/bnxt/bnxt_en/if_bnxt.c +++ b/sys/dev/bnxt/bnxt_en/if_bnxt.c @@ -1978,10 +1978,6 @@ static void bnxt_fw_reset_close(struct bnxt_softc *bp) bnxt_fw_fatal_close(bp); } - iflib_request_reset(bp->ctx); - bnxt_stop(bp->ctx); - bnxt_hwrm_func_drv_unrgtr(bp, false); - for (i = bp->nrxqsets-1; i>=0; i--) { if (BNXT_CHIP_P5_PLUS(bp)) iflib_irq_free(bp->ctx, &bp->nq_rings[i].irq); @@ -2310,10 +2306,52 @@ static bool bnxt_fw_reset_timeout(struct bnxt_softc *bp) (bp->fw_reset_max_dsecs * HZ / 10)); } +/* enqueue an iflib reset and wait for RUNNING/OACTIVE to flip 1->0->1. */ +static int +bnxt_iflib_reset_sync(struct bnxt_softc *bp, int timeout_us) +{ + unsigned flags; + bool reset_started; + int remaining_us = timeout_us; + + /* request reset (async) and enqueue admin task */ + iflib_request_reset(bp->ctx); + iflib_admin_intr_deferred(bp->ctx); + + /* see if reset already started */ + flags = if_getdrvflags(bp->ifp); + reset_started = ((flags & IFF_DRV_RUNNING) == 0) || + ((flags & IFF_DRV_OACTIVE) != 0); + + while (remaining_us > 0) { + flags = if_getdrvflags(bp->ifp); + + if (!reset_started) { + /* wait for reset to START (RUNNING=0 or OACTIVE=1) */ + if ((flags & IFF_DRV_RUNNING) == 0 || (flags & IFF_DRV_OACTIVE) != 0) + reset_started = true; + } else { + /* wait for reset to FINISH (RUNNING=1 and OACTIVE=0) */ + if ((flags & IFF_DRV_RUNNING) != 0 && (flags & IFF_DRV_OACTIVE) == 0) + return 0; /* success */ + } + + pause_sbt("bnxtrst", SBT_1US * 20, 0, 0); /* 20 us sleep */ + remaining_us -= 20; + } + return ETIMEDOUT; +} + static int bnxt_open(struct bnxt_softc *bp) { int rc = 0; + rc = bnxt_hwrm_func_reset(bp); + if (rc) { + device_printf(bp->dev, "reinit: hwrm func_reset failed\n"); + return rc; + } + /* A function/chip reset drops the CAGR tick-resolution register * programmed in bnxt_attach_pre(); restore it before we start * relying on interrupt coalescing again. */ @@ -2380,10 +2418,6 @@ static int bnxt_open(struct bnxt_softc *bp) } bnxt_msix_intr_assign(bp->ctx, 0); - rc = bnxt_init_hw(bp->ctx); - if (rc != 0) - return (rc); - bnxt_intr_enable(bp->ctx); if (test_and_clear_bit(BNXT_STATE_FW_RESET_DET, &bp->state)) { if (!test_bit(BNXT_STATE_IN_FW_RESET, &bp->state)) { @@ -2391,7 +2425,13 @@ static int bnxt_open(struct bnxt_softc *bp) } } - device_printf(bp->dev, "Network interface is UP and operational\n"); + rc = bnxt_iflib_reset_sync(bp, 3000000); + if (rc) { + device_printf(bp->dev, "timeout waiting for reset/init\n"); + return rc; + } + + device_printf(bp->dev, "FW Reset Successfull, Interface is UP and Operational\n"); return rc; } @@ -2488,9 +2528,8 @@ static void bnxt_fw_reset_task(struct work_struct *work) case BNXT_FW_RESET_STATE_OPENING: rc = bnxt_open(bp); if (rc) { - device_printf(bp->dev, "bnxt_open() failed during FW reset\n"); + device_printf(bp->dev, "Firmware reset failed (err=%d)\n", rc); bnxt_fw_reset_abort(bp, rc); - rtnl_unlock(); return; } @@ -3452,7 +3491,8 @@ bnxt_func_reset(struct bnxt_softc *softc) return; } - bnxt_hwrm_resource_free(softc); + if (!test_bit(BNXT_STATE_IN_FW_RESET, &softc->state)) + bnxt_hwrm_resource_free(softc); return; } @@ -5729,7 +5769,8 @@ bnxt_handle_async_event(struct bnxt_softc *softc, struct cmpl_base *cmpl) softc->fw_health->survivals++; set_bit(BNXT_STATE_FW_NON_FATAL_COND, &softc->state); } - device_printf(softc->dev, + device_printf(softc->dev, "%s firmware reset event arrived.\n", type_str); + bnxt_log_live(softc, BNXT_LOGGER_L2, "%s firmware reset event, data1: 0x%x, data2: 0x%x, min wait %u ms, max wait %u ms\n", type_str, data1, data2, softc->fw_reset_min_dsecs * 100,