From nobody Fri Aug 21 00:44:47 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hR1mJ127qz6pPPg for ; Fri, 21 Aug 2026 00:44:48 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hR1mH2hZVz4589 for ; Fri, 21 Aug 2026 00:44:47 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787273087; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=dFEblFJy9VfVfse2ZUUnv44YEvKMcdtyx1rFenEOs8Q=; b=fcNoEmP3kBrGcYxYS0h2aIHXKv5p85Rmy9yb14W5Vq55QWIn7YsGFncg1j5X+sZnGevZTt AGuX3p52Fmngo8bO0+eTQZ1cDChGwanvya4vTmAWRklodUvj7T3VuKcZrdL3nyW9A7KqXn t/k9GsAy/b17SlDXH6Kpqf74kVD/cktumEQHx1OWivyOnEqkO0M2F/q0IX+MxJzy4cBEUv xuaJUTlUEBGk0vLeTO0EDsOLercaBgDCWyqDUhU5eY4pUv1DKgB9/Jta7/uqZE8phR6GKh bBXUQ62QVCxdWlEni4QkTmG6sIVLwgZ/7RBsV9+aXF35PVxo3E9dFJxDkqnwSw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1787273087; a=rsa-sha256; cv=none; b=AMC5fCJtsbgZoW1rB4X7jkXxayGbxNgVx0FOyKiyyZ4/u7+kKw6xxIFzcS4fTSNe6jUZAI gtptvB+MiEpXKLsFq+cZL9jm5fIq4SslB3VLNV528IqWkYcLtADzFUfqXmTDNOwS5wlH3+ DsaA93+mZ5590LQbtBRDHzitg2lFAQ+99BW3FLySzl6SNjn6k+SOeh+s6O561flKIyLJGz Ya+i/FOLm4abJW/+iZrUTb15ayH0Nu+Jz80oJ0F4sdwcvyLeTyDj//buJcgXAlRf0lifRY lUB0lQSDWWGlXpS1eTYQg+aGqx7r/4o5MVapwFOSco5R+M6vp1X1Sdgec4IzAg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787273087; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=dFEblFJy9VfVfse2ZUUnv44YEvKMcdtyx1rFenEOs8Q=; b=NcAZCxRLuhjgEHik3iPYVQIsLvywPUbgBtoggpBrMiR7rO7lxlPWziH45AsIHNCwUc160K Fyp7vTMk4ecm0nVl70D4AfwrgeX86Pll2ETX1JOfqfwnu+BZO+sM/DX+FY1N86t6wUKmjj RXqENz1CIDkCMpsr9XSnR5ZEjcrPOt+JKuWvBKR9KEWgmHjoe4aEznD0V59MgQ5wAdvypN Sqb7zmd75JKOU454LgHoxkUp+39ZE2XApDkqb6bXqYLoiqBUlCWX0Rgdhh1G3Ldw7566ke 205Mxc1Q/RpVCgmIuXRImL5I70xfQfUjq408TZseGQaAPkFSGxZZoP5BLCndCw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hR1mH1b2szhf0 for ; Fri, 21 Aug 2026 00:44:47 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 2363f by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 21 Aug 2026 00:44:47 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kevin Bowling Subject: git: 0280fff5f78a - stable/14 - ixgbe: restart iflib around SR-IOV reconfiguration List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 0280fff5f78aacfc1e9b1db20ace8ba3eee81003 Auto-Submitted: auto-generated Date: Fri, 21 Aug 2026 00:44:47 +0000 Message-Id: <6a879f7f.2363f.f202857@gitrepo.freebsd.org> The branch stable/14 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=0280fff5f78aacfc1e9b1db20ace8ba3eee81003 commit 0280fff5f78aacfc1e9b1db20ace8ba3eee81003 Author: Kevin Bowling AuthorDate: 2026-08-06 11:03:35 +0000 Commit: Kevin Bowling CommitDate: 2026-08-21 00:42:08 +0000 ixgbe: restart iflib around SR-IOV reconfiguration The IOV callback changes the PF pool, virtualization mode, and hardware queue indices while iflib still considers the old queue layout live. Teardown likewise leaves the software pool and mode at their SR-IOV values. Use iflib stop/mutate/restart transactions for both transitions. Disable VF DMA and PCI VF Enable before queue reuse, let outstanding transactions drain, and restore the non-IOV pool and queue indices on teardown. Remove the redundant driver-local pci_iov_detach() wrapper; iflib already performs that check centrally before the driver detach callback. It may be possible to avoid some restart in the future on this hardware pausing DMA and remapping rings but not pursued yet. (cherry picked from commit 86199f1a74abc76c3bb1ed15ccee7df3cabf3d7b) --- sys/dev/ixgbe/if_ix.c | 10 ++-------- sys/dev/ixgbe/if_sriov.c | 36 +++++++++++++++++++++++++----------- sys/dev/ixgbe/ixgbe_sriov.h | 2 -- 3 files changed, 27 insertions(+), 21 deletions(-) diff --git a/sys/dev/ixgbe/if_ix.c b/sys/dev/ixgbe/if_ix.c index e54a6c5d26be..ca7a277dfc16 100644 --- a/sys/dev/ixgbe/if_ix.c +++ b/sys/dev/ixgbe/if_ix.c @@ -293,8 +293,8 @@ static device_method_t ix_methods[] = { DEVMETHOD(device_suspend, iflib_device_suspend), DEVMETHOD(device_resume, iflib_device_resume), #ifdef PCI_IOV - DEVMETHOD(pci_iov_init, iflib_device_iov_init), - DEVMETHOD(pci_iov_uninit, iflib_device_iov_uninit), + DEVMETHOD(pci_iov_init, iflib_device_iov_init_restart), + DEVMETHOD(pci_iov_uninit, iflib_device_iov_uninit_restart), DEVMETHOD(pci_iov_add_vf, iflib_device_iov_add_vf), #endif /* PCI_IOV */ DEVMETHOD_END @@ -3656,16 +3656,10 @@ static int ixgbe_if_detach(if_ctx_t ctx) { struct ixgbe_softc *sc = iflib_get_softc(ctx); - device_t dev = iflib_get_dev(ctx); u32 ctrl_ext; INIT_DEBUGOUT("ixgbe_detach: begin"); - if (ixgbe_pci_iov_detach(dev) != 0) { - device_printf(dev, "SR-IOV in use; detach first.\n"); - return (EBUSY); - } - ixgbe_setup_low_power_mode(ctx); /* let hardware know driver is unloading */ diff --git a/sys/dev/ixgbe/if_sriov.c b/sys/dev/ixgbe/if_sriov.c index 31db2cbb0407..c05b82533081 100644 --- a/sys/dev/ixgbe/if_sriov.c +++ b/sys/dev/ixgbe/if_sriov.c @@ -41,15 +41,6 @@ MALLOC_DEFINE(M_IXGBE_SRIOV, "ix_sriov", "ix SR-IOV allocations"); -/************************************************************************ - * ixgbe_pci_iov_detach - ************************************************************************/ -int -ixgbe_pci_iov_detach(device_t dev) -{ - return pci_iov_detach(dev); -} - /************************************************************************ * ixgbe_define_iov_schemas ************************************************************************/ @@ -1098,7 +1089,6 @@ ixgbe_if_iov_init(if_ctx_t ctx, u16 num_vfs, const nvlist_t *config) ixgbe_init_mbx_params_pf(&sc->hw); sc->feat_en |= IXGBE_FEATURE_SRIOV; - ixgbe_if_init(sc->ctx); return (retval); @@ -1116,7 +1106,8 @@ ixgbe_if_iov_uninit(if_ctx_t ctx) struct ixgbe_hw *hw; struct ixgbe_softc *sc; uint32_t pf_reg, vf_reg; - int i; + int error, i, iov_pos; + u16 iov_ctl; sc = iflib_get_softc(ctx); hw = &sc->hw; @@ -1132,6 +1123,26 @@ ixgbe_if_iov_uninit(if_ctx_t ctx) vf_reg = 0; IXGBE_WRITE_REG(hw, IXGBE_VFRE(vf_reg), 0); IXGBE_WRITE_REG(hw, IXGBE_VFTE(vf_reg), 0); + IXGBE_WRITE_FLUSH(hw); + + /* + * pci_iov(4) normally clears VF Enable after this callback returns, + * but iflib's restart transaction reuses the PF queues first. Disable + * the VFs here and allow outstanding transactions to drain before the + * queue layout changes. + */ + error = pci_find_extcap(sc->dev, PCIZ_SRIOV, &iov_pos); + if (error == 0) { + iov_ctl = pci_read_config(sc->dev, + iov_pos + PCIR_SRIOV_CTL, 2); + iov_ctl &= ~(PCIM_SRIOV_VF_EN | PCIM_SRIOV_VF_MSE); + pci_write_config(sc->dev, iov_pos + PCIR_SRIOV_CTL, + iov_ctl, 2); + pause("ixiov", MAX(1, howmany(hz, 10))); + } else + device_printf(sc->dev, + "could not disable PCI SR-IOV before queue reuse: %d\n", + error); for (i = 0; i < sc->num_vfs; i++) { if (!(sc->vfs[i].flags & IXGBE_VF_ACTIVE)) @@ -1155,6 +1166,9 @@ ixgbe_if_iov_uninit(if_ctx_t ctx) free(sc->vfs, M_IXGBE_SRIOV); sc->vfs = NULL; sc->feat_en &= ~IXGBE_FEATURE_SRIOV; + sc->pool = 0; + sc->iov_mode = IXGBE_NO_VM; + ixgbe_align_all_queue_indices(sc); sc->iov_vfta_valid = false; sc->iov_vlan_promisc = false; (void)ixgbe_clear_vfta(hw); diff --git a/sys/dev/ixgbe/ixgbe_sriov.h b/sys/dev/ixgbe/ixgbe_sriov.h index 84b5e25a8c3c..c38f4075b97a 100644 --- a/sys/dev/ixgbe/ixgbe_sriov.h +++ b/sys/dev/ixgbe/ixgbe_sriov.h @@ -79,7 +79,6 @@ void ixgbe_initialize_iov(struct ixgbe_softc *); void ixgbe_recalculate_max_frame(struct ixgbe_softc *); void ixgbe_ping_all_vfs(struct ixgbe_softc *); u_int ixgbe_iov_rebuild_mta(struct ixgbe_softc *); -int ixgbe_pci_iov_detach(device_t); void ixgbe_define_iov_schemas(device_t, int *); void ixgbe_align_all_queue_indices(struct ixgbe_softc *); int ixgbe_vf_que_index(int, int, int); @@ -96,7 +95,6 @@ u32 ixgbe_get_mrqc(int); #define ixgbe_initialize_iov(_a) #define ixgbe_recalculate_max_frame(_a) #define ixgbe_ping_all_vfs(_a) -#define ixgbe_pci_iov_detach(_a) 0 #define ixgbe_define_iov_schemas(_a,_b) #define ixgbe_align_all_queue_indices(_a) #define ixgbe_vf_que_index(_a, _b, _c) (_c)