From nobody Thu Oct 29 06:40:41 2026 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4jFZP62m2Dz6xtv4 for ; Thu, 29 Oct 2026 06:40:42 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4jFZP553VYz3KTL for ; Thu, 29 Oct 2026 06:40:41 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1793256041; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XQCT54249QD3AqQEVoTd/eIQKGuO/5CCNBzqSJhsscg=; b=TpGwjF2AwTrX98LYlvn/m4/vFwPUva9f1+6XC37BV5qpoC0AuWwm9qQREoDBt9sEpJLz7d Tii48XP+bZfLqw5qw8zYamtcyKbxfIRTXDXIwJQsJbAkbQuvUrLYvOSwwFJnMcTamksu+C eHMWuj/Pwm56JcQN64EPxHIqCMJ+YI7a2YayVMUapIeqP6e13acAI8R2spAWBsW9ug+6Ek 2G6qUTRobkQlCuLkQM2jC/0M6hM2HrchrWNU36VluNQOhPhJELhbtrjOXxhb0fYohHxFpm oXvpydzI7iXmkFTNv2hUa4M1hrmGoOpfRI9uEWQspHi64aGj6r5fpxywDi6/9Q== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1793256041; b=C7TbJ7f7KMFzOP4Fnnlsq6RUrYi39uv+4y3qCEzdmzZ67+dBneJwBA2VWAXGFPmBK2XEoZ xov4JsJQjRlsfiDDOqFqxMjCaprQ9P9fC6nsY8n9U05nvzoNVrvgSr5Uyh9o9Ed2JeFLIM tx6x5GJfWYaH5MMf6Rr5I9EhLWW+Z1Jnu5WBjQEBksezDVwH/48B/VJdVRNVxn6+UYLJ39 Q3Sey/UEANay8Xs8ehfRrTnY8JlQc1zc9tItO74AfpAVh3UBo2wi4oRMTgUTsRqGAuIIyj M7nPZ7R4n7cCpwT4LCL/Y/Z/WSt5AxBw075HD6CT1U5nu3fCNut51RDp6ymwwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1793256041; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XQCT54249QD3AqQEVoTd/eIQKGuO/5CCNBzqSJhsscg=; b=GFNkxbH2xlJzM6xrGTcsOCp4CnmMR5uMVQ1jRnhrpm3g0uX/YmHSGXLYog0qbv0hyM5jo8 PNuOtUcojmooNjq6S/HGyNVjZ1i/jPaAW0w+lAVb+RUJGCFmQ97PvqWpaSXGeWoVIj29Ti U8ui0uuj8ylW4rkbMZlspUtDVaNSP5kUJEZ17JbLGDd1suQUrz97TfoCqUUMWCODtn4AXc w/Qlu2fjS0VDHOgecsjP9NN60AsMf0MVqG5E1dDGXFh3u1xuEoI3uLAlnBDG++dnpfdbmL h1gjCynDy8D4SJ5FuTJibmH57WhWSSvIY11Uu4wf80qB8HJenH5pSNUxz+w97w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4jFZP54PltzRhq for ; Thu, 29 Oct 2026 06:40:41 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3e36b by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Thu, 29 Oct 2026 06:40:41 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kevin Bowling Subject: git: b0c90960f2a4 - stable/15 - ice: Fail closed when VF reset does not complete List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org List-Id: Commits to the stable branches of the FreeBSD src repository List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: b0c90960f2a4c4cc5cba781523e04cb0ea74c273 Auto-Submitted: auto-generated Date: Thu, 29 Oct 2026 06:40:41 +0000 Message-Id: <6ae2ea69.3e36b.2075f8ee@gitrepo.freebsd.org> The branch stable/15 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=b0c90960f2a4c4cc5cba781523e04cb0ea74c273 commit b0c90960f2a4c4cc5cba781523e04cb0ea74c273 Author: Kevin Bowling AuthorDate: 2026-08-19 03:31:43 +0000 Commit: Kevin Bowling CommitDate: 2026-10-29 06:40:32 +0000 ice: Fail closed when VF reset does not complete ice_reset_vf() logs failures to drain PCIe transactions, issue the mandatory zero-queue command, or observe VFR completion, but still publishes VFACTIVE. A VF can resume against reset state which the PF knows is incomplete. Return an error from the reset operation and retain a reset-failed flag when any mandatory stage fails. Reject ordinary virtchnl requests while the failure persists. Publish VFACTIVE only after every stage succeeds. A later VFLR or PF rebuild can recover the VF and clear the failure. Remove tracked queue leaves before clearing their software state. The reset-only AdminQ command drains hardware queues but does not update the shared scheduler database; losing that bookkeeping can strand queue resources across VF teardown and recreation. After a successful VF reset, discard software switch filter state whose hardware rules were reset and clear guest-owned MAC and VLAN tracking. Restore PF-owned anti-spoof policy, the broadcast filter, and the assigned MAC before publishing VFACTIVE. The guest can then replay its own filter state without stale software entries suppressing the firmware requests. Tested on an E810-XXV with a host-attached iavf. Repeated VFLRs rebuilt the base filters with new firmware rule IDs and restored traffic. Forced reset-stage and policy-replay failures remained inactive until recovery, and a PF reset rebuilt the VF and restored live traffic automatically. This follows the conservative reset policy used by ixl(4). Sponsored by: BBOX.io Differential Revision: https://reviews.freebsd.org/D59023 (cherry picked from commit 66d271178feb4a07c774204352228db517c596dc) --- share/man/man4/ice.4 | 4 ++ sys/dev/ice/ice_iov.c | 126 +++++++++++++++++++++++++++++++++++++++++++------- sys/dev/ice/ice_iov.h | 1 + 3 files changed, 114 insertions(+), 17 deletions(-) diff --git a/share/man/man4/ice.4 b/share/man/man4/ice.4 index 9e588548ad95..c6b2ea95c83d 100644 --- a/share/man/man4/ice.4 +++ b/share/man/man4/ice.4 @@ -1158,6 +1158,10 @@ If one VF cannot be reconstructed, it remains configured but uninitialized while the PF and its sibling VFs continue recovering. A VF function-level reset cannot recreate this PF-owned VSI state; another PF or device reset, or SR-IOV configuration recreation, is required. +If a mandatory stage of an individual VF reset fails, the driver likewise +leaves that VF inactive instead of publishing incomplete hardware state. +A later VF reset can retry this recovery; a PF or device reset, or SR-IOV +configuration recreation, can also recover the VF. .Pp An up to date list of parameters and their defaults can be found by using .Xr iovctl 8 diff --git a/sys/dev/ice/ice_iov.c b/sys/dev/ice/ice_iov.c index 57362d8f5a89..518662ccf6c0 100644 --- a/sys/dev/ice/ice_iov.c +++ b/sys/dev/ice/ice_iov.c @@ -56,9 +56,12 @@ SYSCTL_INT(_debug_fail_point_ice_iov, OID_AUTO, vf, static struct ice_vf *ice_iov_get_vf(struct ice_softc *sc, int vf_num); static int ice_iov_configure_mac_anti_spoof(struct ice_softc *sc, struct ice_vf *vf); +static int ice_iov_restore_vf_host_config(struct ice_softc *sc, + struct ice_vf *vf); +static void ice_iov_clear_vf_queue_state(struct ice_vf *vf); static void ice_iov_ready_vf(struct ice_softc *sc, struct ice_vf *vf); -static void ice_reset_vf(struct ice_softc *sc, struct ice_vf *vf, - bool trigger_vflr); +static int ice_reset_vf(struct ice_softc *sc, struct ice_vf *vf, + bool trigger_vflr); static void ice_iov_setup_intr_mapping(struct ice_softc *sc, struct ice_vf *vf); static void ice_vc_version_msg(struct ice_softc *sc, struct ice_vf *vf, @@ -81,6 +84,8 @@ static void ice_vc_enable_queues_msg(struct ice_softc *sc, struct ice_vf *vf, static void ice_vc_notify_vf_link_state(struct ice_softc *sc, struct ice_vf *vf); static void ice_vc_disable_queues_msg(struct ice_softc *sc, struct ice_vf *vf, u8 *msg_buf); +static int ice_vc_disable_queues(struct ice_softc *sc, struct ice_vf *vf, + u32 tx_queues, u32 rx_queues); static void ice_vc_cfg_irq_map_msg(struct ice_softc *sc, struct ice_vf *vf, u8 *msg_buf); static void ice_vc_get_stats_msg(struct ice_softc *sc, struct ice_vf *vf, @@ -250,7 +255,8 @@ ice_iov_get_vf(struct ice_softc *sc, int vf_num) * @vf: VF whose VSI security policy should be configured * * PF and device resets discard the hardware VSI context, so callers must - * replay this policy after creating or rebuilding the VF's VSI. + * replay this policy after creating or rebuilding the VF's VSI. Also reapply + * the PF-owned policy defensively before releasing a VF after VFR. */ static int ice_iov_configure_mac_anti_spoof(struct ice_softc *sc, struct ice_vf *vf) @@ -285,6 +291,43 @@ ice_iov_configure_mac_anti_spoof(struct ice_softc *sc, struct ice_vf *vf) return (0); } +/** + * ice_iov_restore_vf_host_config - Restore PF-owned policy after a VF reset + * @sc: device softc structure + * @vf: VF whose host configuration should be restored + * + * A VF reset discards the guest's filter configuration. Remove the matching + * software switch state as well so that replayed guest requests reach + * firmware instead of being mistaken for filters which still exist. Restore + * the PF-owned source-MAC policy and base filters before releasing the VF. + */ +static int +ice_iov_restore_vf_host_config(struct ice_softc *sc, struct ice_vf *vf) +{ + struct ice_vsi *vsi = vf->vsi; + int error; + + ice_remove_vsi_fltr(&sc->hw, vsi->idx); + vf->mac_filter_cnt = 0; + vf->vlan_cnt = 0; + bzero(vf->vlans_map, sizeof(vf->vlans_map)); + + error = ice_iov_configure_mac_anti_spoof(sc, vf); + if (error != 0) + return (error); + + error = ice_add_vsi_mac_filter(vsi, broadcastaddr); + if (error != 0) + return (error); + if (!ETHER_IS_ZERO(vf->mac)) { + error = ice_add_vsi_mac_filter(vsi, vf->mac); + if (error != 0) + return (error); + } + + return (0); +} + /** * ice_iov_add_vf - Called by the OS for each VF to create * @sc: device softc structure @@ -652,6 +695,18 @@ ice_iov_notify_vfs_reset(struct ice_softc *sc) } } +/** + * ice_iov_clear_vf_queue_state - Clear tracked VF queue state + * @vf: driver's VF structure for the VF to update + */ +static void +ice_iov_clear_vf_queue_state(struct ice_vf *vf) +{ + vf->txq_configured = 0; + vf->rxq_configured = 0; + vf->rxq_enabled = 0; +} + /** * ice_iov_ready_vf - Setup VF interrupts and mark it as ready * @sc: device softc structure @@ -668,9 +723,7 @@ ice_iov_ready_vf(struct ice_softc *sc, struct ice_vf *vf) u32 reg; /* A VF or PF reset discards all queue configuration and state. */ - vf->txq_configured = 0; - vf->rxq_configured = 0; - vf->rxq_enabled = 0; + ice_iov_clear_vf_queue_state(vf); /* Clear the triggering bit */ reg = rd32(hw, VPGEN_VFRTRIG(vf->vf_num)); @@ -708,6 +761,7 @@ ice_iov_rebuild_vf(struct ice_softc *sc, struct ice_vsi *vsi) vf = ice_iov_get_vf(sc, vsi->vf_num); atomic_clear_32(&vf->vf_flags, VF_FLAG_INITIALIZED); atomic_set_32(&vf->vf_flags, VF_FLAG_REBUILD_FAILED); + ice_iov_clear_vf_queue_state(vf); ICE_IOV_FAIL_POINT(sc, vf->vf_num, rebuild_before_initialize, error, fail); @@ -734,7 +788,8 @@ ice_iov_rebuild_vf(struct ice_softc *sc, struct ice_vsi *vsi) return (EIO); } - atomic_clear_32(&vf->vf_flags, VF_FLAG_REBUILD_FAILED); + atomic_clear_32(&vf->vf_flags, + VF_FLAG_REBUILD_FAILED | VF_FLAG_RESET_FAILED); ice_iov_ready_vf(sc, vf); return (0); @@ -750,24 +805,26 @@ fail: * @vf: driver's VF structure for VF to be reset * @trigger_vflr: trigger a reset or only handle already executed reset * - * Performs a VFR for the given VF. This function busy waits until the - * reset completes in the HW, notifies the VF that the reset is done - * by setting a bit in a HW register, then returns. + * Performs a VFR for the given VF. This function busy waits until the reset + * completes in the HW and publishes VFACTIVE only after every mandatory + * reset stage succeeds. * * @remark This also sets up the PF<->VF interrupt mapping and allocations in * the hardware after the hardware reset is finished, via * ice_iov_setup_intr_mapping() */ -static void +static int ice_reset_vf(struct ice_softc *sc, struct ice_vf *vf, bool trigger_vflr) { u16 global_vf_num, reg_idx, bit_idx; struct ice_hw *hw = &sc->hw; - int status; + int error, status; u32 reg; int i; global_vf_num = vf->vf_num + hw->func_caps.vf_base_id; + atomic_clear_32(&vf->vf_flags, VF_FLAG_INITIALIZED); + error = 0; if (trigger_vflr) { reg = rd32(hw, VPGEN_VFRTRIG(vf->vf_num)); @@ -792,18 +849,36 @@ ice_reset_vf(struct ice_softc *sc, struct ice_vf *vf, bool trigger_vflr) DELAY(ICE_PCI_CIAD_WAIT_DELAY_US); } - if (i == ICE_PCI_CIAD_WAIT_COUNT) + if (i == ICE_PCI_CIAD_WAIT_COUNT) { device_printf(sc->dev, "VF-%d PCI transactions stuck\n", vf->vf_num); + error = ETIMEDOUT; + } + + /* + * Remove the tracked queue leaves from the software scheduler before + * issuing the reset-only AQ command. That command drains hardware but + * does not update the shared scheduler database. Retain unresolved queue + * state if cleanup fails so a later reset can retry it. + */ + status = ice_vc_disable_queues(sc, vf, vf->txq_configured, + vf->rxq_enabled); + if (status == 0) + ice_iov_clear_vf_queue_state(vf); + else if (error == 0) + error = status; /* This zero-queue command is required to complete every VF reset. */ status = ice_dis_vsi_txq(hw->port_info, vf->vsi->idx, 0, 0, NULL, NULL, NULL, ICE_VF_RESET, vf->vf_num, NULL); - if (status) + if (status) { device_printf(sc->dev, "%s: Failed to disable LAN Tx queues: err %s aq_err %s\n", __func__, ice_status_str(status), ice_aq_str(hw->adminq.sq_last_status)); + if (error == 0) + error = EIO; + } /* Then check for the VF reset to finish in HW */ for (i = 0; i < ICE_VPGEN_VFRSTAT_WAIT_COUNT; i++) { @@ -813,16 +888,33 @@ ice_reset_vf(struct ice_softc *sc, struct ice_vf *vf, bool trigger_vflr) DELAY(ICE_VPGEN_VFRSTAT_WAIT_DELAY_US); } - if (i == ICE_VPGEN_VFRSTAT_WAIT_COUNT) + if (i == ICE_VPGEN_VFRSTAT_WAIT_COUNT) { device_printf(sc->dev, "VF-%d Reset is stuck\n", vf->vf_num); + if (error == 0) + error = ETIMEDOUT; + } + + if (error != 0) { + atomic_set_32(&vf->vf_flags, VF_FLAG_RESET_FAILED); + return (error); + } + + atomic_clear_32(&vf->vf_flags, VF_FLAG_RESET_FAILED); /* A VFR cannot recover PF-owned VSI state lost during PF rebuild. */ if ((atomic_load_acq_32(&vf->vf_flags) & VF_FLAG_REBUILD_FAILED) != 0) - return; + return (EIO); + + error = ice_iov_restore_vf_host_config(sc, vf); + if (error != 0) { + atomic_set_32(&vf->vf_flags, VF_FLAG_RESET_FAILED); + return (error); + } ice_iov_ready_vf(sc, vf); + return (0); } /** @@ -2317,7 +2409,7 @@ ice_vc_handle_vf_msg(struct ice_softc *sc, struct ice_rq_event_info *event) return; /* Only a later PF rebuild can restore an invalid firmware VSI. */ - if ((vf_flags & VF_FLAG_REBUILD_FAILED) != 0 && + if ((vf_flags & (VF_FLAG_REBUILD_FAILED | VF_FLAG_RESET_FAILED)) != 0 && v_opcode != VIRTCHNL_OP_VERSION && v_opcode != VIRTCHNL_OP_RESET_VF) { ice_aq_send_msg_to_vf(hw, v_id, v_opcode, diff --git a/sys/dev/ice/ice_iov.h b/sys/dev/ice/ice_iov.h index 0ebd1b1c0555..795c190dfc50 100644 --- a/sys/dev/ice/ice_iov.h +++ b/sys/dev/ice/ice_iov.h @@ -72,6 +72,7 @@ enum ice_vf_flags { VF_FLAG_MAC_ANTI_SPOOF = BIT(4), VF_FLAG_INITIALIZED = BIT(5), VF_FLAG_REBUILD_FAILED = BIT(6), + VF_FLAG_RESET_FAILED = BIT(7), }; /**