From nobody Fri Aug 21 00:30:52 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hR1SF0033z6pMnx for ; Fri, 21 Aug 2026 00:30:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hR1SD5W5Yz3x5m for ; Fri, 21 Aug 2026 00:30:52 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787272252; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=X6dLJPkO3QfDIrJBEJGIzGzbBXKwvteVQG1djCUzcJU=; b=ZDFCfC3rGlcvF3erkbmV2Vn/1LX82ZF0dNu7GCOjs4S9Auu5HF2Jm67e1SJV/Q7lK7OHpd 72SB1LgEP5TCQcfRva6UF9MPMFGEtz0ufSWimD776J6yqaQEJyeUsvS4/tX5HTJyDeej8i xdYyFt6SNzJ35rMSfL+ViI4NzWpkOgwwLQnRqexELsKAfbu4zrxPxDvqwLDKf4qxd99+eE 0HL9RQNlPQtT4URWDyOeBCdlhyA83cVlIXWWZOfcPVK+czbjn5iysQVr8r+6awQz1rzhBL +B+MKKSokFwmdjT8pZp3A8rpwAEwb2qYs9BXFsc/1HYJVh/8ta4hUSf8WAOZew== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1787272252; a=rsa-sha256; cv=none; b=haLySTeDWIgK4KxFCkawJwVbb/QzhIvw7AFq8GoM3EydnajofIaxRhUDAfnMx5jyk7rTGg e6EbtK593ck3O2CK4HayB1Q/UYvH+wR6pnN1Mvj3ySGASRNa7wupqlZGPIue7tVjn8mr5F 2fZZyxSlAnyKtWE3xmaPodLGdn1ka0gqp2PfFXQbphv/z4Rew5QRIouYeorNaEaNSYTpyI M9IPbZeCWfOvmxBK9KtVlwYgdOWOrdXDEdM+EK0fjLOLwPwPPx1irO6WFmPQ+KEtwc/4Dd aJqnY0KndvAulVOvlhigxrX7oAHuTpShXkovuV6FP2CTXr6j/x8m8/2eY8XFsg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787272252; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=X6dLJPkO3QfDIrJBEJGIzGzbBXKwvteVQG1djCUzcJU=; b=HF0oL6gKHAQgFMVHVVuy4zYFayyJNyONSrt4GdYPwAGLEP2CIwtZG5mISV93uQBwkEpo08 6u/YK17xbFzqLG7qjYl/rJ/9d0B6epnwgj8UcAhiHqICBloM81AK88ofsWDR30xzILpxur 11lXPkTfkfaGnaQ2sQDeqM8FoTwQh/gyb5NUdXZ3syBayuJnqkh/L6R1F6zUWQKbIZsv+f IygrbCTXtB6x+L3+CyJacdOE67cLw52Ddrg6R6MaNCE43jo6mP6H7ptsFy3zAJOUC53Lfu l8DhuQ8QuRVWnj+8meVPvN6jt/hMzi7mZLYvMUhCNzWAfz+nJSwTT9wy+h+1Ig== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hR1SD49cVzh44 for ; Fri, 21 Aug 2026 00:30:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 21eb9 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 21 Aug 2026 00:30:52 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kevin Bowling Subject: git: 57a732245fd1 - stable/15 - ixgbe: restart iflib around SR-IOV reconfiguration List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 57a732245fd1e67d4fc1fcf6a411e80a0381bdc0 Auto-Submitted: auto-generated Date: Fri, 21 Aug 2026 00:30:52 +0000 Message-Id: <6a879c3c.21eb9.6a8c53c3@gitrepo.freebsd.org> The branch stable/15 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=57a732245fd1e67d4fc1fcf6a411e80a0381bdc0 commit 57a732245fd1e67d4fc1fcf6a411e80a0381bdc0 Author: Kevin Bowling AuthorDate: 2026-08-06 11:03:35 +0000 Commit: Kevin Bowling CommitDate: 2026-08-21 00:30:39 +0000 ixgbe: restart iflib around SR-IOV reconfiguration The IOV callback changes the PF pool, virtualization mode, and hardware queue indices while iflib still considers the old queue layout live. Teardown likewise leaves the software pool and mode at their SR-IOV values. Use iflib stop/mutate/restart transactions for both transitions. Disable VF DMA and PCI VF Enable before queue reuse, let outstanding transactions drain, and restore the non-IOV pool and queue indices on teardown. Remove the redundant driver-local pci_iov_detach() wrapper; iflib already performs that check centrally before the driver detach callback. It may be possible to avoid some restart in the future on this hardware pausing DMA and remapping rings but not pursued yet. (cherry picked from commit 86199f1a74abc76c3bb1ed15ccee7df3cabf3d7b) --- sys/dev/ixgbe/if_ix.c | 10 ++-------- sys/dev/ixgbe/if_sriov.c | 36 +++++++++++++++++++++++++----------- sys/dev/ixgbe/ixgbe_sriov.h | 2 -- 3 files changed, 27 insertions(+), 21 deletions(-) diff --git a/sys/dev/ixgbe/if_ix.c b/sys/dev/ixgbe/if_ix.c index e54a6c5d26be..ca7a277dfc16 100644 --- a/sys/dev/ixgbe/if_ix.c +++ b/sys/dev/ixgbe/if_ix.c @@ -293,8 +293,8 @@ static device_method_t ix_methods[] = { DEVMETHOD(device_suspend, iflib_device_suspend), DEVMETHOD(device_resume, iflib_device_resume), #ifdef PCI_IOV - DEVMETHOD(pci_iov_init, iflib_device_iov_init), - DEVMETHOD(pci_iov_uninit, iflib_device_iov_uninit), + DEVMETHOD(pci_iov_init, iflib_device_iov_init_restart), + DEVMETHOD(pci_iov_uninit, iflib_device_iov_uninit_restart), DEVMETHOD(pci_iov_add_vf, iflib_device_iov_add_vf), #endif /* PCI_IOV */ DEVMETHOD_END @@ -3656,16 +3656,10 @@ static int ixgbe_if_detach(if_ctx_t ctx) { struct ixgbe_softc *sc = iflib_get_softc(ctx); - device_t dev = iflib_get_dev(ctx); u32 ctrl_ext; INIT_DEBUGOUT("ixgbe_detach: begin"); - if (ixgbe_pci_iov_detach(dev) != 0) { - device_printf(dev, "SR-IOV in use; detach first.\n"); - return (EBUSY); - } - ixgbe_setup_low_power_mode(ctx); /* let hardware know driver is unloading */ diff --git a/sys/dev/ixgbe/if_sriov.c b/sys/dev/ixgbe/if_sriov.c index 31db2cbb0407..c05b82533081 100644 --- a/sys/dev/ixgbe/if_sriov.c +++ b/sys/dev/ixgbe/if_sriov.c @@ -41,15 +41,6 @@ MALLOC_DEFINE(M_IXGBE_SRIOV, "ix_sriov", "ix SR-IOV allocations"); -/************************************************************************ - * ixgbe_pci_iov_detach - ************************************************************************/ -int -ixgbe_pci_iov_detach(device_t dev) -{ - return pci_iov_detach(dev); -} - /************************************************************************ * ixgbe_define_iov_schemas ************************************************************************/ @@ -1098,7 +1089,6 @@ ixgbe_if_iov_init(if_ctx_t ctx, u16 num_vfs, const nvlist_t *config) ixgbe_init_mbx_params_pf(&sc->hw); sc->feat_en |= IXGBE_FEATURE_SRIOV; - ixgbe_if_init(sc->ctx); return (retval); @@ -1116,7 +1106,8 @@ ixgbe_if_iov_uninit(if_ctx_t ctx) struct ixgbe_hw *hw; struct ixgbe_softc *sc; uint32_t pf_reg, vf_reg; - int i; + int error, i, iov_pos; + u16 iov_ctl; sc = iflib_get_softc(ctx); hw = &sc->hw; @@ -1132,6 +1123,26 @@ ixgbe_if_iov_uninit(if_ctx_t ctx) vf_reg = 0; IXGBE_WRITE_REG(hw, IXGBE_VFRE(vf_reg), 0); IXGBE_WRITE_REG(hw, IXGBE_VFTE(vf_reg), 0); + IXGBE_WRITE_FLUSH(hw); + + /* + * pci_iov(4) normally clears VF Enable after this callback returns, + * but iflib's restart transaction reuses the PF queues first. Disable + * the VFs here and allow outstanding transactions to drain before the + * queue layout changes. + */ + error = pci_find_extcap(sc->dev, PCIZ_SRIOV, &iov_pos); + if (error == 0) { + iov_ctl = pci_read_config(sc->dev, + iov_pos + PCIR_SRIOV_CTL, 2); + iov_ctl &= ~(PCIM_SRIOV_VF_EN | PCIM_SRIOV_VF_MSE); + pci_write_config(sc->dev, iov_pos + PCIR_SRIOV_CTL, + iov_ctl, 2); + pause("ixiov", MAX(1, howmany(hz, 10))); + } else + device_printf(sc->dev, + "could not disable PCI SR-IOV before queue reuse: %d\n", + error); for (i = 0; i < sc->num_vfs; i++) { if (!(sc->vfs[i].flags & IXGBE_VF_ACTIVE)) @@ -1155,6 +1166,9 @@ ixgbe_if_iov_uninit(if_ctx_t ctx) free(sc->vfs, M_IXGBE_SRIOV); sc->vfs = NULL; sc->feat_en &= ~IXGBE_FEATURE_SRIOV; + sc->pool = 0; + sc->iov_mode = IXGBE_NO_VM; + ixgbe_align_all_queue_indices(sc); sc->iov_vfta_valid = false; sc->iov_vlan_promisc = false; (void)ixgbe_clear_vfta(hw); diff --git a/sys/dev/ixgbe/ixgbe_sriov.h b/sys/dev/ixgbe/ixgbe_sriov.h index 84b5e25a8c3c..c38f4075b97a 100644 --- a/sys/dev/ixgbe/ixgbe_sriov.h +++ b/sys/dev/ixgbe/ixgbe_sriov.h @@ -79,7 +79,6 @@ void ixgbe_initialize_iov(struct ixgbe_softc *); void ixgbe_recalculate_max_frame(struct ixgbe_softc *); void ixgbe_ping_all_vfs(struct ixgbe_softc *); u_int ixgbe_iov_rebuild_mta(struct ixgbe_softc *); -int ixgbe_pci_iov_detach(device_t); void ixgbe_define_iov_schemas(device_t, int *); void ixgbe_align_all_queue_indices(struct ixgbe_softc *); int ixgbe_vf_que_index(int, int, int); @@ -96,7 +95,6 @@ u32 ixgbe_get_mrqc(int); #define ixgbe_initialize_iov(_a) #define ixgbe_recalculate_max_frame(_a) #define ixgbe_ping_all_vfs(_a) -#define ixgbe_pci_iov_detach(_a) 0 #define ixgbe_define_iov_schemas(_a,_b) #define ixgbe_align_all_queue_indices(_a) #define ixgbe_vf_que_index(_a, _b, _c) (_c)