From nobody Mon Aug 24 00:53:43 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hSsqD18qQz6q2Rx for ; Mon, 24 Aug 2026 00:53:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hSsqC3wVYz3vtD for ; Mon, 24 Aug 2026 00:53:43 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787532823; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BpCYOTOwCJA8ML6kVqggevzE68d0Q8jFygQn7+qOHGw=; b=BHjZeTEgExe2ZOQkBIO9lcn9cE/v8f2xUvPbXJpi9Vf6EfHDS7VX1qkcwdZUAXNm9DuYvL Qxn4tUyDDBtsCMG455a5liCDDW9m5vQLawXmP5ej0zYJnpGF+zja4oUPetl0SBkqQqnAqg 3LQNrvQL9uzHfxWoPLfJ3tDLptnXdMGhuIssAsy5Mnm9YC39wvREnyBF9cPyPxqtbRhTg3 3C8+k4Yc5vTXRxGqY+VFSbE6Wu0EpCDwfPeY2nFsuH1J+c5EiXtUaiPmR7QTWb+pSOa9zJ WeOlF24tAEOMSivg3dh41Z3z2ets6LnohpewEeXX3/WHxNTSxShWLcmpPgbKmQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1787532823; a=rsa-sha256; cv=none; b=J5IaGvNXDz7KliexcjP2HOeA/jwRjmSQCEobelGZoPv3WYX223REe/3kWm3LORTAMki8Qb Ln5sYPUZeOEoA0gjcBMpkVjr4ukHUIOuuz8STrCDXj2EC4r6YgGf5ZzJzHBKswDCKfZ/pK 8H+//Jy00uJHVSu5GTwxigxJSxaATWGPJwpusDGidUj82Nr1v/BdKyuTrPaPCzm4pRSYGs LgKWlOjrfh2fgAgYgehu/YoVvA/+kgmpPnfUGb16zIsH6kDdFZjFOSdUA3o/SFivdA+Mgw QmsdRtMCiDDtw0sQC8jy96qI0l2c6I7WbjNWBNF/DVD57AhK4n4444DSb5FMDA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787532823; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BpCYOTOwCJA8ML6kVqggevzE68d0Q8jFygQn7+qOHGw=; b=BPGNSt0SQc6Vxweid8dd5kpLGZhArTVx7v5B8u1eL/VXPNwUxBh6W3rJicKjm1IugfZfGJ 70TlE7Q95Vvw5KRfxsC8wUwp3Do+csngWWq6D7wWDfGSZZ+yph+tSJV+eKx/LfDLaNhzDs TFrnUZjxz4KAFrixx0ycFwGNdY4YRwVqkzREmku15tllJUdjCTFR1CCKbTgeaPFDonZqRn Jiva/XdrOZ716zKs4qRNqfYZ6fVcm+ld8AshoaxGJNcDKFlj+dFFvCuLOF1qYNA15vZZ5u BR+xxUstqJClde20S9lABxXrv11U0VejwqyBRK/fAmfwmecvO+0pkKimruXhQg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hSsqC30F1zhxl for ; Mon, 24 Aug 2026 00:53:43 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 321e7 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Mon, 24 Aug 2026 00:53:43 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kevin Bowling Subject: git: 1319574637fb - stable/15 - ixl: Quiesce VF DMA before a PF reset List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 1319574637fb15f3ce176ce1638e25db05340c9a Auto-Submitted: auto-generated Date: Mon, 24 Aug 2026 00:53:43 +0000 Message-Id: <6a8b9617.321e7.64416d5e@gitrepo.freebsd.org> The branch stable/15 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=1319574637fb15f3ce176ce1638e25db05340c9a commit 1319574637fb15f3ce176ce1638e25db05340c9a Author: Kevin Bowling AuthorDate: 2026-08-10 11:56:57 +0000 Commit: Kevin Bowling CommitDate: 2026-08-24 00:49:57 +0000 ixl: Quiesce VF DMA before a PF reset A PF reset has a warning interval before the hardware reset begins. Cooperative VF drivers respond to the reset event by stopping and releasing their receive buffers, but notifying VFs did not stop the hardware queues. An active VF could therefore DMA through its old rings into freed mbuf clusters during the warning interval. Put every enabled VF in reset, drain its PCIe transactions, disable its queues, wait for receive queue shutdown, and drain transactions again before tearing down the PF HMC and AdminQ. Hold VFs in reset again while rebuilding the firmware topology. Release VF reset before programming the replacement VSI and queue mappings, since VF reset clears those registers, and publish VFACTIVE only after reconstruction succeeds. Leave a VF held in reset if rebuilding it fails. Fixes: 983e628a0c47 ("ixl: Rebuild VF resources after a PF reset") (cherry picked from commit 0048dfddc049f3ef050ba16544f42503d5c42db8) --- sys/dev/ixl/ixl_pf.h | 1 + sys/dev/ixl/ixl_pf_iflib.c | 16 ++++-- sys/dev/ixl/ixl_pf_iov.c | 122 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 136 insertions(+), 3 deletions(-) diff --git a/sys/dev/ixl/ixl_pf.h b/sys/dev/ixl/ixl_pf.h index e79fcac3c4c8..93595c551998 100644 --- a/sys/dev/ixl/ixl_pf.h +++ b/sys/dev/ixl/ixl_pf.h @@ -391,6 +391,7 @@ int ixl_pf_reset(struct ixl_pf *); #ifdef PCI_IOV void ixl_notify_vfs_reset(struct ixl_pf *); +int ixl_quiesce_vfs_for_reset(struct ixl_pf *); int ixl_rebuild_vfs_after_reset(struct ixl_pf *); int ixl_reset_vf_on_mdd(struct ixl_pf *, uint16_t); #endif diff --git a/sys/dev/ixl/ixl_pf_iflib.c b/sys/dev/ixl/ixl_pf_iflib.c index 1f48519344f7..bdacf0cfa06d 100644 --- a/sys/dev/ixl/ixl_pf_iflib.c +++ b/sys/dev/ixl/ixl_pf_iflib.c @@ -941,10 +941,17 @@ ixl_prepare_for_reset(struct ixl_pf *pf, bool is_up) { struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; - int error = 0; + int error, first_error; + first_error = 0; #ifdef PCI_IOV ixl_notify_vfs_reset(pf); + error = ixl_quiesce_vfs_for_reset(pf); + if (error != 0) { + device_printf(dev, "Failed to quiesce one or more VFs: %d\n", + error); + first_error = error; + } #endif if (is_up) ixl_if_stop(pf->vsi.ctx); @@ -954,12 +961,15 @@ ixl_prepare_for_reset(struct ixl_pf *pf, bool is_up) ixl_disable_intr0(hw); error = i40e_shutdown_adminq(hw); - if (error) + if (error) { device_printf(dev, "Shutdown Admin queue failed with code %d\n", error); + if (first_error == 0) + first_error = error; + } ixl_pf_qmgr_release(&pf->qmgr, &pf->qtag); - return (error); + return (first_error); } int diff --git a/sys/dev/ixl/ixl_pf_iov.c b/sys/dev/ixl/ixl_pf_iov.c index 765f4d17682a..ae6ae16ac56d 100644 --- a/sys/dev/ixl/ixl_pf_iov.c +++ b/sys/dev/ixl/ixl_pf_iov.c @@ -2063,6 +2063,117 @@ ixl_reset_vf_on_mdd(struct ixl_pf *pf, uint16_t vfnum) return (ixl_reset_vf(pf, vf)); } +/* + * Hold every configured VF in reset and drain its PCIe transactions. + * Starting all VF resets before polling amortizes the hardware reset delay + * across the set. This is used both before the device reset and while its + * replacement firmware topology is constructed. + */ +static int +ixl_hold_vfs_in_reset(struct ixl_pf *pf) +{ + struct i40e_hw *hw; + struct ixl_vf *vf; + uint32_t vfrstat, vfrtrig; + int error, first_error, i, retry; + bool all_done; + + hw = &pf->hw; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vf->vf_flags &= ~VF_FLAG_INITIALIZED; + vfrtrig = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num)); + vfrtrig |= I40E_VPGEN_VFRTRIG_VFSWR_MASK; + wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num), vfrtrig); + } + ixl_flush(hw); + + first_error = 0; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_flush_pcie(pf, vf); + if (error != 0) { + device_printf(pf->dev, + "Timed out draining PCIe transactions on VF-%d\n", i); + if (first_error == 0) + first_error = error; + } + } + + for (retry = 0; retry < IXL_VF_RESET_TIMEOUT; retry++) { + DELAY(1000); + all_done = true; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vfrstat = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_num)); + if (!(vfrstat & I40E_VPGEN_VFRSTAT_VFRD_MASK)) { + all_done = false; + break; + } + } + if (all_done) + return (first_error); + } + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vfrstat = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_num)); + if (!(vfrstat & I40E_VPGEN_VFRSTAT_VFRD_MASK)) + device_printf(pf->dev, + "VF-%d failed to enter reset\n", i); + } + return (first_error != 0 ? first_error : ETIMEDOUT); +} + +/* + * Stop VF DMA during the reset-warning interval, before VF drivers may + * release their buffers. Holding VFR also prevents an uncooperative VF + * from re-enabling a queue before the device reset begins. + */ +int +ixl_quiesce_vfs_for_reset(struct ixl_pf *pf) +{ + struct ixl_vf *vf; + int error, first_error, i; + + if (pf->num_vfs == 0) + return (0); + first_error = ixl_hold_vfs_in_reset(pf); + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_disable_rings(pf, &vf->vsi, &vf->qtag); + if (error != 0 && first_error == 0) + first_error = error; + } + /* Hardware may need up to 50 ms to finish disabling RX queues. */ + DELAY(50000); + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_flush_pcie(pf, vf); + if (error != 0) { + device_printf(pf->dev, + "PCIe transactions remain pending on VF-%d\n", i); + if (first_error == 0) + first_error = error; + } + } + return (first_error); +} + int ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) { @@ -2073,6 +2184,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) hw = &pf->hw; pf->veb_seid = 0; + /* The reset-warning path already stopped and drained the old rings. */ + error = ixl_hold_vfs_in_reset(pf); for (i = 0; i < pf->num_vfs; i++) { vf = &pf->vfs[i]; if (!(vf->vf_flags & VF_FLAG_ENABLED)) @@ -2096,6 +2209,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) vf->mdd_event_pending = false; vf->mdd_reset_pending = false; } + if (error != 0) + return (error); error = ixl_setup_iov_switch(pf); if (error != 0) @@ -2107,6 +2222,11 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) if (!(vf->vf_flags & VF_FLAG_ENABLED)) continue; + /* + * VFR resets the VF queue-mapping registers. Release it before + * programming the replacement VSI and mappings, but do not publish + * VFACTIVE until reconstruction is complete. + */ wr32(hw, I40E_VFGEN_RSTAT1(vf->vf_num), VIRTCHNL_VFR_COMPLETED); vfrtrig = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num)); @@ -2117,6 +2237,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) if (error != 0) { device_printf(pf->dev, "Failed to rebuild VF-%d: %d\n", i, error); + vfrtrig |= I40E_VPGEN_VFRTRIG_VFSWR_MASK; + wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num), vfrtrig); if (first_error == 0) first_error = error; continue;