From nobody Thu Oct 29 06:40:09 2026 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4jFZNT3xvyz6xv1k for ; Thu, 29 Oct 2026 06:40:09 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4jFZNT1fY5z3JtC for ; Thu, 29 Oct 2026 06:40:09 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1793256009; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=xOl6/hBqBjMvY7OpmIhf+iEpcjxNKzmT7T3bFPUol54=; b=RegyItk3vg2Jjm+NtJqlmU6qMAllr+40ecp18P6i82wJ0L8vQWLh79W6wtRmDPk97SJw/1 S6z41+ZZGqspzZ13jbaLNrQ5K5vf+jMNoRBzeFMGKW0TKse3D9c1tLjtNYh7RjEh4NZ+f3 E++ZoMneRh//4wvVZOHvKs6MxuIHPUz4XHYeV49t3puVnvWAoSRk6K49Eeaxyfqp8B1+kv nT8LEd6ZTADUaMylzpY20NSnS16yrRru2PJGdzAZm12cm0aFElJj/APzEteVCgo81945q6 g1bg+0DCZPOt2RPnhq1NC56VuJGvGwPq0BzTEtUNMwL0PyfTviTfDSnAMd3AdA== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1793256009; b=f6H3dP5W7PGiodvW4tMx0KLl3PlXc2nKRyZJeSaV6D86+cexT5kcpMEZcbg6VijOLD9Rmi 6+qGm9X7APZGrbnPE+e162iRy1jtpzFtdTA10H0COGh6Fkkti0BjZXJ+QjHCbUWSDn7MmX 6pSp9/KI7XWl5V0lBxfmzAxVfUtxiPeZTvvhcYnQmZv4LRgcqG6+39kSSdyAJwHt97oH+g xztkJMa1XGPyI7fl4AoeCB6lL/eIF2HHOCr+r4k59sAh1LrkFrZ/dMZ0Vr/M2u0NvnOjBE rYjajlsk53DDVfiRYqz5lwq5N63IpE8ErdcO6Nmv7qhq9eQqeu04ottgX+7LUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1793256009; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=xOl6/hBqBjMvY7OpmIhf+iEpcjxNKzmT7T3bFPUol54=; b=Ti9Xc9OJ1bQLb8dQq0n4cb4LRntYEZmQ494xFUlGzMSyEQJzTnJyC+4RDs93s6LexNTu8u TE6OHAOtr1erhZVE7Bq31jhv5mSZ4F7oPkM3Hz7S6B0ERl1QKjQpTNR7yZLxxaA9eQb00Z bIUN5isbHKgQkkPiBI7vqImKML5TAA0n8JwOyAaxuzoXT4cECgXr177Y5C/AcTMEh7Ha7p Y28IAsCbNTLv2Js9ngg+eOqeyPu+qbkSRAPsSB0wJc0enwKjwXn1VhGxr0WlLIQCC0iHWM KTN7+yG9OJX+3R8CsB32kKboWSSwsrQ4cuKclanZdolXNDVNfXhS/tSrd//RXQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4jFZNT1JnYzRmP for ; Thu, 29 Oct 2026 06:40:09 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3dcd3 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Thu, 29 Oct 2026 06:40:09 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kevin Bowling Subject: git: 9755b23f7f8c - stable/15 - ice: Fix SR-IOV VF resource cleanup List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org List-Id: Commits to the stable branches of the FreeBSD src repository List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 9755b23f7f8cbbff32d529b591ab90881f8e9427 Auto-Submitted: auto-generated Date: Thu, 29 Oct 2026 06:40:09 +0000 Message-Id: <6ae2ea49.3dcd3.1411495@gitrepo.freebsd.org> The branch stable/15 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=9755b23f7f8cbbff32d529b591ab90881f8e9427 commit 9755b23f7f8cbbff32d529b591ab90881f8e9427 Author: Kevin Bowling AuthorDate: 2026-08-18 08:48:36 +0000 Commit: Kevin Bowling CommitDate: 2026-10-29 06:39:13 +0000 ice: Fix SR-IOV VF resource cleanup ice_iov_uninit() freed each VF interrupt-map array without returning the reserved indices to the device interrupt resource manager. Repeated VF create and destroy cycles therefore exhausted the PF interrupt map even though no VFs remained. Return the interrupt allocation before freeing its map. Also split software-only VSI release from hardware teardown so failures before ice_initialize_vsi() do not issue invalid RSS, scheduler, and Free VSI commands for an object firmware has never seen. Keep a VF disabled until all of its resources and hardware state have been created successfully. Clear the enabled state before teardown and after any failed add so asynchronous mailbox processing cannot use a partial or freed VSI. Consume VFLR status for inactive VF slots without trying to reset a nonexistent VSI. Track whether firmware currently owns each VSI and clear that ownership after resets. Teardown can then skip AdminQ commands for VSIs which were not rebuilt. Remove every VSI switch filter before firmware teardown, matching Linux and preventing filter-list leaks across create and destroy cycles. This also applies to the PF VSI detach path. Validated on an E810-XXV with two consecutive create and destroy cycles of 128 four-queue VFs. A 16-queue VF could then be created. Two oversized configurations each failed, cleaned back to zero VFs without invalid firmware teardown commands, and were each followed by a successful 16-queue VF creation. Sponsored by: BBOX.io Differential Revision: https://reviews.freebsd.org/D58908 (cherry picked from commit 468384c8bd1130a880fd87ac5c2d182123f9f6ec) --- sys/dev/ice/ice_iov.c | 28 ++++++++++++--- sys/dev/ice/ice_lib.c | 85 ++++++++++++++++++++++++++++++++-------------- sys/dev/ice/ice_lib.h | 2 ++ sys/dev/ice/if_ice_iflib.c | 6 ++++ 4 files changed, 91 insertions(+), 30 deletions(-) diff --git a/sys/dev/ice/ice_iov.c b/sys/dev/ice/ice_iov.c index 0d53f8f14253..869fe43a9411 100644 --- a/sys/dev/ice/ice_iov.c +++ b/sys/dev/ice/ice_iov.c @@ -227,7 +227,7 @@ ice_iov_add_vf(struct ice_softc *sc, uint16_t vfnum, const nvlist_t *params) int i; vf = ice_iov_get_vf(sc, vfnum); - vf->vf_flags = VF_FLAG_ENABLED; + vf->vf_flags = 0; /* This VF needs at least one VSI */ vsi = ice_alloc_vsi(sc, ICE_VSI_VF); @@ -385,6 +385,7 @@ ice_iov_add_vf(struct ice_softc *sc, uint16_t vfnum, const nvlist_t *params) goto release_imap; } + atomic_set_32(&vf->vf_flags, VF_FLAG_ENABLED); ice_iov_ready_vf(sc, vf); return (0); @@ -408,8 +409,12 @@ free_txqs: free(vsi->tx_queues, M_ICE); vsi->tx_queues = NULL; release_vsi: - ice_release_vsi(vsi); + if (vsi->hw_vsi_created) + ice_release_vsi(vsi); + else + ice_release_vsi_resources(vsi); vf->vsi = NULL; + atomic_store_rel_32(&vf->vf_flags, 0); return (error); } @@ -426,10 +431,13 @@ ice_iov_uninit(struct ice_softc *sc) /* Release per-VF resources */ for (int i = 0; i < sc->num_vfs; i++) { vf = &sc->vfs[i]; + atomic_store_rel_32(&vf->vf_flags, 0); vsi = vf->vsi; /* Free VF interrupt reservation */ if (vf->vf_imap) { + ice_resmgr_release_map(&sc->dev_imgr, vf->vf_imap, + vf->num_irq_vectors); free(vf->vf_imap, M_ICE); vf->vf_imap = NULL; } @@ -457,7 +465,10 @@ ice_iov_uninit(struct ice_softc *sc) vsi->rx_queues = NULL; } - ice_release_vsi(vsi); + if (vsi->hw_vsi_created) + ice_release_vsi(vsi); + else + ice_release_vsi_resources(vsi); vf->vsi = NULL; } @@ -489,8 +500,17 @@ ice_iov_handle_vflr(struct ice_softc *sc) reg_idx = (hw->func_caps.vf_base_id + vf->vf_num) / 32; bit_idx = (hw->func_caps.vf_base_id + vf->vf_num) % 32; reg = rd32(hw, GLGEN_VFLRSTAT(reg_idx)); - if (reg & BIT(bit_idx)) + if ((reg & BIT(bit_idx)) == 0) + continue; + if ((atomic_load_acq_32(&vf->vf_flags) & + VF_FLAG_ENABLED) != 0 && vf->vsi != NULL) { ice_reset_vf(sc, vf, false); + continue; + } + + /* Consume reset events for inactive or incompletely added VFs. */ + wr32(hw, GLGEN_VFLRSTAT(reg_idx), BIT(bit_idx)); + ice_flush(hw); } } diff --git a/sys/dev/ice/ice_lib.c b/sys/dev/ice/ice_lib.c index 8e593358fafe..56a3ce0e465f 100644 --- a/sys/dev/ice/ice_lib.c +++ b/sys/dev/ice/ice_lib.c @@ -777,6 +777,7 @@ ice_initialize_vsi(struct ice_vsi *vsi) ice_aq_str(hw->adminq.sq_last_status)); return (EIO); } + vsi->hw_vsi_created = true; vsi->info = ctx.info; /* Initialize VSI with just 1 TC to start */ @@ -816,6 +817,8 @@ ice_deinit_vsi(struct ice_vsi *vsi) /* Assert that the VSI pointer matches in the list */ MPASS(vsi == sc->all_vsi[vsi->idx]); + if (!vsi->hw_vsi_created) + return; ctx.info = vsi->info; @@ -837,9 +840,32 @@ ice_deinit_vsi(struct ice_vsi *vsi) "Free VSI %u AQ call failed, err %s aq_err %s\n", vsi->idx, ice_status_str(status), ice_aq_str(hw->adminq.sq_last_status)); + } else { + vsi->hw_vsi_created = false; } } +/* + * Release the queue maps and storage owned by a VSI. Callers must remove + * the VSI sysctl context before reaching this helper. + */ +static void +ice_free_vsi_resources(struct ice_vsi *vsi) +{ + struct ice_softc *sc = vsi->sc; + int idx = vsi->idx; + + /* Assert that the VSI pointer matches in the list */ + MPASS(vsi == sc->all_vsi[idx]); + + ice_free_vsi_qmaps(vsi); + + if (vsi->dynamic) + free(sc->all_vsi[idx], M_ICE); + + sc->all_vsi[idx] = NULL; +} + /** * ice_release_vsi - Release resources associated with a VSI * @vsi: the VSI to release @@ -851,35 +877,39 @@ ice_deinit_vsi(struct ice_vsi *vsi) void ice_release_vsi(struct ice_vsi *vsi) { - struct ice_softc *sc = vsi->sc; - int idx = vsi->idx; - - /* Assert that the VSI pointer matches in the list */ - MPASS(vsi == sc->all_vsi[idx]); + MPASS(vsi == vsi->sc->all_vsi[vsi->idx]); /* Cleanup RSS configuration */ - if (ice_is_bit_set(sc->feat_en, ICE_FEATURE_RSS)) + if (ice_is_bit_set(vsi->sc->feat_en, ICE_FEATURE_RSS)) ice_clean_vsi_rss_cfg(vsi); + /* Drain sysctl handlers before invalidating the hardware VSI. */ ice_del_vsi_sysctl_ctx(vsi); - /* Remove the configured mirror rule, if it exists */ - ice_remove_vsi_mirroring(vsi); - - /* - * If we unload the driver after a reset fails, we do not need to do - * this step. - */ - if (!ice_test_state(&sc->state, ICE_STATE_RESET_FAILED)) + /* Do not issue firmware commands for a missing VSI or failed device. */ + if (vsi->hw_vsi_created && + !ice_test_state(&vsi->sc->state, ICE_STATE_RESET_FAILED)) { + ice_remove_vsi_mirroring(vsi); + ice_remove_vsi_fltr(&vsi->sc->hw, vsi->idx); ice_deinit_vsi(vsi); - - ice_free_vsi_qmaps(vsi); - - if (vsi->dynamic) { - free(sc->all_vsi[idx], M_ICE); } - sc->all_vsi[idx] = NULL; + ice_free_vsi_resources(vsi); +} + +/** + * ice_release_vsi_resources - Release software resources for a VSI + * @vsi: the VSI to release + * + * Release resources allocated by ice_alloc_vsi() without issuing firmware + * commands. This is used when setup fails before ice_initialize_vsi() has + * attempted to create the VSI in hardware. + */ +void +ice_release_vsi_resources(struct ice_vsi *vsi) +{ + ice_del_vsi_sysctl_ctx(vsi); + ice_free_vsi_resources(vsi); } /** @@ -7908,13 +7938,16 @@ ice_clean_vsi_rss_cfg(struct ice_vsi *vsi) device_t dev = sc->dev; int status; - status = ice_rem_vsi_rss_cfg(hw, vsi->idx); - if (status) - device_printf(dev, - "Failed to remove RSS configuration for VSI %d, err %s\n", - vsi->idx, ice_status_str(status)); + if (vsi->hw_vsi_created && + !ice_test_state(&sc->state, ICE_STATE_RESET_FAILED)) { + status = ice_rem_vsi_rss_cfg(hw, vsi->idx); + if (status) + device_printf(dev, + "Failed to remove RSS configuration for VSI %d, err %s\n", + vsi->idx, ice_status_str(status)); + } - /* Remove this VSI from the RSS list */ + /* Remove software tracking even if the hardware VSI no longer exists. */ ice_rem_vsi_rss_list(hw, vsi->idx); } diff --git a/sys/dev/ice/ice_lib.h b/sys/dev/ice/ice_lib.h index 2562a9e2476f..6c6f93b97228 100644 --- a/sys/dev/ice/ice_lib.h +++ b/sys/dev/ice/ice_lib.h @@ -558,6 +558,7 @@ struct ice_vsi { struct ice_softc *sc; bool dynamic; /* if true, dynamically allocated */ + bool hw_vsi_created; /* firmware owns a VSI for this handle */ enum ice_vsi_type type; /* type of this VSI */ u16 idx; /* software index to sc->all_vsi[] */ @@ -935,6 +936,7 @@ int ice_map_bar(device_t dev, struct ice_bar_info *bar, int bar_num); void ice_free_bar(device_t dev, struct ice_bar_info *bar); void ice_set_ctrlq_len(struct ice_hw *hw); void ice_release_vsi(struct ice_vsi *vsi); +void ice_release_vsi_resources(struct ice_vsi *vsi); struct ice_vsi *ice_alloc_vsi(struct ice_softc *sc, enum ice_vsi_type type); void ice_alloc_vsi_qmap(struct ice_vsi *vsi, const int max_tx_queues, const int max_rx_queues); diff --git a/sys/dev/ice/if_ice_iflib.c b/sys/dev/ice/if_ice_iflib.c index 80c752473720..2989d5968dae 100644 --- a/sys/dev/ice/if_ice_iflib.c +++ b/sys/dev/ice/if_ice_iflib.c @@ -2696,11 +2696,17 @@ ice_rebuild(struct ice_softc *sc) enum ice_ddp_state pkg_state; int status; int err; + int i; sc->rebuild_ticks = ticks; /* If we're rebuilding, then a reset has succeeded. */ ice_clear_state(&sc->state, ICE_STATE_RESET_FAILED); + /* The reset discarded every firmware VSI before reconstruction. */ + for (i = 0; i < sc->num_available_vsi; i++) { + if (sc->all_vsi[i] != NULL) + sc->all_vsi[i]->hw_vsi_created = false; + } /* * If the firmware is in recovery mode, only restore the limited