From nobody Mon Aug 24 09:14:21 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hT4wt0qvXz6p6Pn for ; Mon, 24 Aug 2026 09:14:22 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hT4ws6N4lz3vsm for ; Mon, 24 Aug 2026 09:14:21 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787562861; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=kppHgk1Rb3DkKtKnD2KoncLAs3sKGaoSiQvbQo8F3wg=; b=deCQDfLS/AWImqkHCdAmLL0HpTqFVcyCUZuZ8JEnZ/XtPA+JzABXBF0PrstsE6729yrc8v I1ZLsd2xa0LwWbfPDx4oolJ9GdKXMQkVTiDRekSD16HCUtmu1VYIs4epfdtdTjDztfTTQG fruv4HbzBOAzzx+BbLUI8GKri4po6Nt5KEI3Vk7Epg4FECf+1jG2da+nZnBK4yxuRGtl/p 7E+j5qymXJuT1upLyLh3YKIk2PFJxvb5cZAziAJuWtFUclQ9KKTtkqi8cJVqnLnGcJoaLc smvVDCvP+I+SNjp/EqOZo5febmw2VU/I3er5Vp2oemzplL2YPNcdyDOQfzcYOw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1787562861; a=rsa-sha256; cv=none; b=mCFRWACaCPXbvL7QIdLT056Prt4iFDCjR1g4VXlO9AFBymND7FX+hJxB6DZQZf+b7ULhuV 5dRt2IXTt3cKCVtA612yjRM8b1uLYOTJ9LdaS3xzZA0u9/nw5u5PJJpiFBKTGK6HQy4Ois gyCGeFThn8Acn9X7GHSbpJAoab+xvw/A2Qs5NQkC+v+wN/A6zJcJFriGzLiY+eK1jnZA2Y /E13zRyBjkhREbFTNf2ZYZPmpBUXILniNOiBj1/PuXvkNAJMnUt1d9SixpaeWht3bdCo94 lPOVc9n+6sknALfK+QhzqKKl5EIec6E9weBi9+9EFtKZ3Xxk5oJcb/60EA/fqw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787562861; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=kppHgk1Rb3DkKtKnD2KoncLAs3sKGaoSiQvbQo8F3wg=; b=PtWXFBOYZmwvyDydZ0ifB7pQc99NXhFAQQolhjoqQBnQQfsUx53zziliqv+ZmANMfmJfKK qWf9FYs253mhUJIFqiHqofHwhKY7WyJK0NyXZIVCpwNSjNITaA37kaE9mLTQSZeKx4yqKX juvRtYVwIvUn3xVCGT+rDQM8HUorqWqgskLftQ+/jVnQH/9EoEhFs3b9moFQZWvuC9vaFI qcMSCcK8Hc7k75Q65qf39gzFDg9MwO2PG/IlMClQHHBVjKylK8jvQ7tmXlBfA9svZ33REm 21M3iTaMjLHOqyeXpbE6AL8Lri/ID5uwUqM8D0k0cL6JERgDIiUCySXwb1LXEw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hT4ws59JCzyR7 for ; Mon, 24 Aug 2026 09:14:21 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 47f7d by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Mon, 24 Aug 2026 09:14:21 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kevin Bowling Subject: git: a9f39c3bc2ec - stable/14 - ixl: Quiesce VF DMA before a PF reset List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: a9f39c3bc2ecfa39ea6fc8a08a459b32ed1743c6 Auto-Submitted: auto-generated Date: Mon, 24 Aug 2026 09:14:21 +0000 Message-Id: <6a8c0b6d.47f7d.597efad7@gitrepo.freebsd.org> The branch stable/14 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=a9f39c3bc2ecfa39ea6fc8a08a459b32ed1743c6 commit a9f39c3bc2ecfa39ea6fc8a08a459b32ed1743c6 Author: Kevin Bowling AuthorDate: 2026-08-10 11:56:57 +0000 Commit: Kevin Bowling CommitDate: 2026-08-24 09:12:22 +0000 ixl: Quiesce VF DMA before a PF reset A PF reset has a warning interval before the hardware reset begins. Cooperative VF drivers respond to the reset event by stopping and releasing their receive buffers, but notifying VFs did not stop the hardware queues. An active VF could therefore DMA through its old rings into freed mbuf clusters during the warning interval. Put every enabled VF in reset, drain its PCIe transactions, disable its queues, wait for receive queue shutdown, and drain transactions again before tearing down the PF HMC and AdminQ. Hold VFs in reset again while rebuilding the firmware topology. Release VF reset before programming the replacement VSI and queue mappings, since VF reset clears those registers, and publish VFACTIVE only after reconstruction succeeds. Leave a VF held in reset if rebuilding it fails. Fixes: 983e628a0c47 ("ixl: Rebuild VF resources after a PF reset") (cherry picked from commit 0048dfddc049f3ef050ba16544f42503d5c42db8) --- sys/dev/ixl/ixl_pf.h | 1 + sys/dev/ixl/ixl_pf_iflib.c | 16 ++++-- sys/dev/ixl/ixl_pf_iov.c | 121 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 135 insertions(+), 3 deletions(-) diff --git a/sys/dev/ixl/ixl_pf.h b/sys/dev/ixl/ixl_pf.h index f7bcf121683f..02d455c806a7 100644 --- a/sys/dev/ixl/ixl_pf.h +++ b/sys/dev/ixl/ixl_pf.h @@ -389,6 +389,7 @@ int ixl_pf_reset(struct ixl_pf *); #ifdef PCI_IOV void ixl_notify_vfs_reset(struct ixl_pf *); +int ixl_quiesce_vfs_for_reset(struct ixl_pf *); int ixl_rebuild_vfs_after_reset(struct ixl_pf *); int ixl_reset_vf_on_mdd(struct ixl_pf *, uint16_t); #endif diff --git a/sys/dev/ixl/ixl_pf_iflib.c b/sys/dev/ixl/ixl_pf_iflib.c index 1f48519344f7..bdacf0cfa06d 100644 --- a/sys/dev/ixl/ixl_pf_iflib.c +++ b/sys/dev/ixl/ixl_pf_iflib.c @@ -941,10 +941,17 @@ ixl_prepare_for_reset(struct ixl_pf *pf, bool is_up) { struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; - int error = 0; + int error, first_error; + first_error = 0; #ifdef PCI_IOV ixl_notify_vfs_reset(pf); + error = ixl_quiesce_vfs_for_reset(pf); + if (error != 0) { + device_printf(dev, "Failed to quiesce one or more VFs: %d\n", + error); + first_error = error; + } #endif if (is_up) ixl_if_stop(pf->vsi.ctx); @@ -954,12 +961,15 @@ ixl_prepare_for_reset(struct ixl_pf *pf, bool is_up) ixl_disable_intr0(hw); error = i40e_shutdown_adminq(hw); - if (error) + if (error) { device_printf(dev, "Shutdown Admin queue failed with code %d\n", error); + if (first_error == 0) + first_error = error; + } ixl_pf_qmgr_release(&pf->qmgr, &pf->qtag); - return (error); + return (first_error); } int diff --git a/sys/dev/ixl/ixl_pf_iov.c b/sys/dev/ixl/ixl_pf_iov.c index 560b559f40e2..2ecbed5bea14 100644 --- a/sys/dev/ixl/ixl_pf_iov.c +++ b/sys/dev/ixl/ixl_pf_iov.c @@ -2063,6 +2063,116 @@ ixl_reset_vf_on_mdd(struct ixl_pf *pf, uint16_t vfnum) return (ixl_reset_vf(pf, vf)); } +/* + * Hold every configured VF in reset and drain its PCIe transactions. + * Starting all VF resets before polling amortizes the hardware reset delay + * across the set. This is used both before the device reset and while its + * replacement firmware topology is constructed. + */ +static int +ixl_hold_vfs_in_reset(struct ixl_pf *pf) +{ + struct i40e_hw *hw; + struct ixl_vf *vf; + uint32_t vfrstat, vfrtrig; + int error, first_error, i, retry; + bool all_done; + + hw = &pf->hw; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vfrtrig = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num)); + vfrtrig |= I40E_VPGEN_VFRTRIG_VFSWR_MASK; + wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num), vfrtrig); + } + ixl_flush(hw); + + first_error = 0; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_flush_pcie(pf, vf); + if (error != 0) { + device_printf(pf->dev, + "Timed out draining PCIe transactions on VF-%d\n", i); + if (first_error == 0) + first_error = error; + } + } + + for (retry = 0; retry < IXL_VF_RESET_TIMEOUT; retry++) { + DELAY(1000); + all_done = true; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vfrstat = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_num)); + if (!(vfrstat & I40E_VPGEN_VFRSTAT_VFRD_MASK)) { + all_done = false; + break; + } + } + if (all_done) + return (first_error); + } + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vfrstat = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_num)); + if (!(vfrstat & I40E_VPGEN_VFRSTAT_VFRD_MASK)) + device_printf(pf->dev, + "VF-%d failed to enter reset\n", i); + } + return (first_error != 0 ? first_error : ETIMEDOUT); +} + +/* + * Stop VF DMA during the reset-warning interval, before VF drivers may + * release their buffers. Holding VFR also prevents an uncooperative VF + * from re-enabling a queue before the device reset begins. + */ +int +ixl_quiesce_vfs_for_reset(struct ixl_pf *pf) +{ + struct ixl_vf *vf; + int error, first_error, i; + + if (pf->num_vfs == 0) + return (0); + first_error = ixl_hold_vfs_in_reset(pf); + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_disable_rings(pf, &vf->vsi, &vf->qtag); + if (error != 0 && first_error == 0) + first_error = error; + } + /* Hardware may need up to 50 ms to finish disabling RX queues. */ + DELAY(50000); + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_flush_pcie(pf, vf); + if (error != 0) { + device_printf(pf->dev, + "PCIe transactions remain pending on VF-%d\n", i); + if (first_error == 0) + first_error = error; + } + } + return (first_error); +} + int ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) { @@ -2073,6 +2183,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) hw = &pf->hw; pf->veb_seid = 0; + /* The reset-warning path already stopped and drained the old rings. */ + error = ixl_hold_vfs_in_reset(pf); for (i = 0; i < pf->num_vfs; i++) { vf = &pf->vfs[i]; if (!(vf->vf_flags & VF_FLAG_ENABLED)) @@ -2095,6 +2207,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) vf->mdd_event_pending = false; vf->mdd_reset_pending = false; } + if (error != 0) + return (error); error = ixl_setup_iov_switch(pf); if (error != 0) @@ -2106,6 +2220,11 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) if (!(vf->vf_flags & VF_FLAG_ENABLED)) continue; + /* + * VFR resets the VF queue-mapping registers. Release it before + * programming the replacement VSI and mappings, but do not publish + * VFACTIVE until reconstruction is complete. + */ wr32(hw, I40E_VFGEN_RSTAT1(vf->vf_num), VIRTCHNL_VFR_COMPLETED); vfrtrig = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num)); @@ -2116,6 +2235,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) if (error != 0) { device_printf(pf->dev, "Failed to rebuild VF-%d: %d\n", i, error); + vfrtrig |= I40E_VPGEN_VFRTRIG_VFSWR_MASK; + wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num), vfrtrig); if (first_error == 0) first_error = error; continue;