From nobody Mon Aug 10 14:00:13 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hJbxB2W6Jz6nq7G for ; Mon, 10 Aug 2026 14:00:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hJbx96qD5z3Q4H for ; Mon, 10 Aug 2026 14:00:13 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1786370414; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=VacDXCuAVjW4IhmqUSaWthRfrBf8qdaLwDh3DBb2+fE=; b=e0294PHWnLrGP5O/I/PzrbzBRfJdKH+VXZjQBi5hBVsN72QDJ23eaDceuiEyhPfs3lmr5O sn7BajfLyzAH/F7l8O+DK5iMHF8I+k4ZfAzC69ooZb3oWFw+9rhY4kzREullTGVCqb015H uaowy4+goVoaVDVf4K6VLHQUf/aK1VI+amKvAYVWIyfXh25cK/vS1gB4mnbvZ3FFhGEf03 BLUFxGBLo2kTmf5SyxgTbGYpVzK+89jZraBbh8z/W1SeQl2/IdhpCNaSQozZT2d/FCGYsz ac0MQgDCgUmEky0d/IYGA7nhhiiLp27nhKOQGajxtNYuGVAefRC3lAhClUxX8Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1786370414; a=rsa-sha256; cv=none; b=Uossrox51BfXbvzi2as0Y3r6vq+z6VQGGEKyZxX+0e2c0rvoptWWZfr3kgE54mdrBQ9rQG P6KFIbQw1RMhLTGCrwZ6xoY0TYaeE9mhYiVcilc9VBGUmoq5Ozc/A7wOJzKagu0GAXcBDz Cfk7aIXG2dPsB5kfYynXiA0D8fg94jjXpM8VigzYo6i0oMgladQ/ougK9tf1vFEvERFYtJ A/BV4jMX+MYclD1kfcx3BlV/4I9DYSSFO1rSPF8OOsa+DZ34eL0rBIHoPeC9Bax6vU5eJN hZMJmKwxzgcDGqhhHq8Gczz0dmdEZ0zHODiT97eJSLFIauVB0nK2SYyP6j3pRA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1786370414; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=VacDXCuAVjW4IhmqUSaWthRfrBf8qdaLwDh3DBb2+fE=; b=l8B9CEEoKdTupAu/aLGOLQVYb4HwY6Pb3P89+wWMNLV7zr+dMRrvBy+BkVGAkq2mFKCa8Y wcttdnZcWrF2JrKegucDEfZUfSR8O75vgku54e7SZd/liOYRhyotjCKpLTGo5ehjR+CA2b Wc3d4mwiJWp4u2GBHarXtNxNdRxGKToHbDeQDVa+cUrvztTB9HOWLl06pKzGXQQdHzMrcH sElZTFmwQ6Q5xpkcUDJSPmulUTJShkFietti3qsNnx8ioVnFsjks4tdUxF1lZ/OgshreST AkS7OfLZrF/eUHOgL8a884IrGujWi5dRgKhO43rbE1sIPZbOM/47R+IUR5O/4w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hJbx95ggCz19Vj for ; Mon, 10 Aug 2026 14:00:13 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 2112c by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Mon, 10 Aug 2026 14:00:13 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Kevin Bowling Subject: git: 0048dfddc049 - main - ixl: Quiesce VF DMA before a PF reset List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 0048dfddc049f3ef050ba16544f42503d5c42db8 Auto-Submitted: auto-generated Date: Mon, 10 Aug 2026 14:00:13 +0000 Message-Id: <6a79d96d.2112c.3ac61eb3@gitrepo.freebsd.org> The branch main has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=0048dfddc049f3ef050ba16544f42503d5c42db8 commit 0048dfddc049f3ef050ba16544f42503d5c42db8 Author: Kevin Bowling AuthorDate: 2026-08-10 11:56:57 +0000 Commit: Kevin Bowling CommitDate: 2026-08-10 13:59:59 +0000 ixl: Quiesce VF DMA before a PF reset A PF reset has a warning interval before the hardware reset begins. Cooperative VF drivers respond to the reset event by stopping and releasing their receive buffers, but notifying VFs did not stop the hardware queues. An active VF could therefore DMA through its old rings into freed mbuf clusters during the warning interval. Put every enabled VF in reset, drain its PCIe transactions, disable its queues, wait for receive queue shutdown, and drain transactions again before tearing down the PF HMC and AdminQ. Hold VFs in reset again while rebuilding the firmware topology. Release VF reset before programming the replacement VSI and queue mappings, since VF reset clears those registers, and publish VFACTIVE only after reconstruction succeeds. Leave a VF held in reset if rebuilding it fails. Fixes: 983e628a0c47 ("ixl: Rebuild VF resources after a PF reset") MFC after: 2 weeks --- sys/dev/ixl/ixl_pf.h | 1 + sys/dev/ixl/ixl_pf_iflib.c | 16 ++++-- sys/dev/ixl/ixl_pf_iov.c | 122 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 136 insertions(+), 3 deletions(-) diff --git a/sys/dev/ixl/ixl_pf.h b/sys/dev/ixl/ixl_pf.h index 80631e861310..74e1cad79731 100644 --- a/sys/dev/ixl/ixl_pf.h +++ b/sys/dev/ixl/ixl_pf.h @@ -392,6 +392,7 @@ int ixl_pf_reset(struct ixl_pf *); #ifdef PCI_IOV void ixl_notify_vfs_reset(struct ixl_pf *); +int ixl_quiesce_vfs_for_reset(struct ixl_pf *); int ixl_rebuild_vfs_after_reset(struct ixl_pf *); int ixl_reset_vf_on_mdd(struct ixl_pf *, uint16_t); #endif diff --git a/sys/dev/ixl/ixl_pf_iflib.c b/sys/dev/ixl/ixl_pf_iflib.c index 1f48519344f7..bdacf0cfa06d 100644 --- a/sys/dev/ixl/ixl_pf_iflib.c +++ b/sys/dev/ixl/ixl_pf_iflib.c @@ -941,10 +941,17 @@ ixl_prepare_for_reset(struct ixl_pf *pf, bool is_up) { struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; - int error = 0; + int error, first_error; + first_error = 0; #ifdef PCI_IOV ixl_notify_vfs_reset(pf); + error = ixl_quiesce_vfs_for_reset(pf); + if (error != 0) { + device_printf(dev, "Failed to quiesce one or more VFs: %d\n", + error); + first_error = error; + } #endif if (is_up) ixl_if_stop(pf->vsi.ctx); @@ -954,12 +961,15 @@ ixl_prepare_for_reset(struct ixl_pf *pf, bool is_up) ixl_disable_intr0(hw); error = i40e_shutdown_adminq(hw); - if (error) + if (error) { device_printf(dev, "Shutdown Admin queue failed with code %d\n", error); + if (first_error == 0) + first_error = error; + } ixl_pf_qmgr_release(&pf->qmgr, &pf->qtag); - return (error); + return (first_error); } int diff --git a/sys/dev/ixl/ixl_pf_iov.c b/sys/dev/ixl/ixl_pf_iov.c index 5254a59df3ab..64e7a5e36bcc 100644 --- a/sys/dev/ixl/ixl_pf_iov.c +++ b/sys/dev/ixl/ixl_pf_iov.c @@ -2065,6 +2065,117 @@ ixl_reset_vf_on_mdd(struct ixl_pf *pf, uint16_t vfnum) return (ixl_reset_vf(pf, vf)); } +/* + * Hold every configured VF in reset and drain its PCIe transactions. + * Starting all VF resets before polling amortizes the hardware reset delay + * across the set. This is used both before the device reset and while its + * replacement firmware topology is constructed. + */ +static int +ixl_hold_vfs_in_reset(struct ixl_pf *pf) +{ + struct i40e_hw *hw; + struct ixl_vf *vf; + uint32_t vfrstat, vfrtrig; + int error, first_error, i, retry; + bool all_done; + + hw = &pf->hw; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vf->vf_flags &= ~VF_FLAG_INITIALIZED; + vfrtrig = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num)); + vfrtrig |= I40E_VPGEN_VFRTRIG_VFSWR_MASK; + wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num), vfrtrig); + } + ixl_flush(hw); + + first_error = 0; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_flush_pcie(pf, vf); + if (error != 0) { + device_printf(pf->dev, + "Timed out draining PCIe transactions on VF-%d\n", i); + if (first_error == 0) + first_error = error; + } + } + + for (retry = 0; retry < IXL_VF_RESET_TIMEOUT; retry++) { + DELAY(1000); + all_done = true; + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vfrstat = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_num)); + if (!(vfrstat & I40E_VPGEN_VFRSTAT_VFRD_MASK)) { + all_done = false; + break; + } + } + if (all_done) + return (first_error); + } + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + vfrstat = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_num)); + if (!(vfrstat & I40E_VPGEN_VFRSTAT_VFRD_MASK)) + device_printf(pf->dev, + "VF-%d failed to enter reset\n", i); + } + return (first_error != 0 ? first_error : ETIMEDOUT); +} + +/* + * Stop VF DMA during the reset-warning interval, before VF drivers may + * release their buffers. Holding VFR also prevents an uncooperative VF + * from re-enabling a queue before the device reset begins. + */ +int +ixl_quiesce_vfs_for_reset(struct ixl_pf *pf) +{ + struct ixl_vf *vf; + int error, first_error, i; + + if (pf->num_vfs == 0) + return (0); + first_error = ixl_hold_vfs_in_reset(pf); + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_disable_rings(pf, &vf->vsi, &vf->qtag); + if (error != 0 && first_error == 0) + first_error = error; + } + /* Hardware may need up to 50 ms to finish disabling RX queues. */ + DELAY(50000); + + for (i = 0; i < pf->num_vfs; i++) { + vf = &pf->vfs[i]; + if (!(vf->vf_flags & VF_FLAG_ENABLED)) + continue; + error = ixl_flush_pcie(pf, vf); + if (error != 0) { + device_printf(pf->dev, + "PCIe transactions remain pending on VF-%d\n", i); + if (first_error == 0) + first_error = error; + } + } + return (first_error); +} + int ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) { @@ -2075,6 +2186,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) hw = &pf->hw; pf->veb_seid = 0; + /* The reset-warning path already stopped and drained the old rings. */ + error = ixl_hold_vfs_in_reset(pf); for (i = 0; i < pf->num_vfs; i++) { vf = &pf->vfs[i]; if (!(vf->vf_flags & VF_FLAG_ENABLED)) @@ -2098,6 +2211,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) vf->mdd_event_pending = false; vf->mdd_reset_pending = false; } + if (error != 0) + return (error); error = ixl_setup_iov_switch(pf); if (error != 0) @@ -2109,6 +2224,11 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) if (!(vf->vf_flags & VF_FLAG_ENABLED)) continue; + /* + * VFR resets the VF queue-mapping registers. Release it before + * programming the replacement VSI and mappings, but do not publish + * VFACTIVE until reconstruction is complete. + */ wr32(hw, I40E_VFGEN_RSTAT1(vf->vf_num), VIRTCHNL_VFR_COMPLETED); vfrtrig = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num)); @@ -2119,6 +2239,8 @@ ixl_rebuild_vfs_after_reset(struct ixl_pf *pf) if (error != 0) { device_printf(pf->dev, "Failed to rebuild VF-%d: %d\n", i, error); + vfrtrig |= I40E_VPGEN_VFRTRIG_VFSWR_MASK; + wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num), vfrtrig); if (first_error == 0) first_error = error; continue;