From nobody Fri Jun 20 13:52:15 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bNzS00sqyz5y449; Fri, 20 Jun 2025 13:52:16 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bNzRz4gn0z3XJp; Fri, 20 Jun 2025 13:52:15 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1750427535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WUg6uMqU5EzeE4O7/EzTN65dCCeIaBD/K7wUoFd+klk=; b=Y9+EfCRS08jXHxcstuPojYU8e+7cpuSPEM4P2a6fG97ypexMc0fRiddOvXnQcydw7a0YMM ATSpZh7hjkUv+wSk/HpZHp3wOf0aqRdDFB05NRiHO2mf5WgJ9XZ/3iKCMmd20ZyN3oeAPn 24O+iGxmyhgcWMU8LHOotIjLkr/w3zy/MRkSC++FWXawJXf8jDreanboWz1oXyTV7CApfO kSU0488h+hgXXEWW0fD2ra7rn/ZhY/05Rds3VytQwtT/RfY7Uq5jDWlBsvV1y7CXPmNFSJ 1p7IRlhaWFA4dTpLEkIPvt1ANjelAlD8i5bw549ImVaJlzPQOOaQbfwIp6yabg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1750427535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WUg6uMqU5EzeE4O7/EzTN65dCCeIaBD/K7wUoFd+klk=; b=WYVqdxL6MGMc2ci4Xy5y2RDmc+eCdv8s5rPfff3W/m/TysPdP9+wr6/SX04wsMlDE2E33v MgKvCJzze6mj45yVvEUUE/xCbfZYMeEhhKG7VQUExqdzD/H5j4TXc1PccqJX1eOtBXGJhe J54pan1aRMcyuFYbEuHxNm9/2gw4/+6+N5hjSJwBzvZTPHnHxEp+jz/t7o6+c8D6gMzQSX +vGjWNTmpwYKcHbs4oWuDK/0NO4yGpQO0yYf4WpDWxDbVXjINNIoVW3D/cAwRCCyhAICNN 39oYu+jw8Ito5+v4bDqFP1UH0oTL6y7GV/zfEN4aEgwXRjZhyo8F1LCGE3Nvkg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1750427535; a=rsa-sha256; cv=none; b=NtIFOrk3m8deh9oL4cCYw2Awk51e/HUQwZVPn8MNCHPiur822DtohPTTrLJarlib8dv/Xy qegqNj7jJb7cksSrH3vA7Qy3BzTbGKEZhKkIg2xTAdXGnVw8DYHpipKdjWkuk7KBLYuAOb 2WnlVn4wLmqcRwyhADnsDS8AapGIaYS+SazpAbjjT6uWJZkbyhMTb7v9WNxoq5njFPVajQ HdR0wW+sn0XEZyRL+VJqbSTtF6p5Dbcwc0olZoJGqk9ntl3w0Ru4PhpE33S5rN27YiDaeB a3/Bj+I7k4pDOQ36vO+/9zgay1kC85Zxu1RX7Otal/5r4j4a4btMvH0lsSbKfw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4bNzRz49d5znCp; Fri, 20 Jun 2025 13:52:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 55KDqFc0057808; Fri, 20 Jun 2025 13:52:15 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 55KDqFdY057805; Fri, 20 Jun 2025 13:52:15 GMT (envelope-from git) Date: Fri, 20 Jun 2025 13:52:15 GMT Message-Id: <202506201352.55KDqFdY057805@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 4e8531ec2633 - stable/14 - gve: Use load-acquire to fetch generation bits List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 4e8531ec2633ea101fa29c09acf73f48cf2bcc17 Auto-Submitted: auto-generated The branch stable/14 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=4e8531ec2633ea101fa29c09acf73f48cf2bcc17 commit 4e8531ec2633ea101fa29c09acf73f48cf2bcc17 Author: Jasper Tran O'Leary AuthorDate: 2025-05-20 23:34:44 +0000 Commit: Mark Johnston CommitDate: 2025-06-20 12:46:09 +0000 gve: Use load-acquire to fetch generation bits When running the driver using the DQO queue format, we must load the generation bit and check it before possibly reading the rest of the descriptor's fields. Previously, we guarded against reordering of reads using an explicit thread fence. This commit changes the thread fence to a load with acquire semantics. Because the tx and rx generation fields are in a bitfield, we cannot explicitly address them in an atomic load. Instead we load the respective containing bytes in the descriptor and mask them appropriately. Signed-off-by: Jasper Tran O'Leary Reviewed by: markj MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D50384 (cherry picked from commit b044f125373e05c20ca16edfd29cccad1f4025bd) --- sys/dev/gve/gve_adminq.h | 3 ++- sys/dev/gve/gve_dqo.h | 20 ++++++++++++++++---- sys/dev/gve/gve_rx_dqo.c | 24 +++++++++++++++++------- sys/dev/gve/gve_tx_dqo.c | 25 +++++++++++++++++-------- 4 files changed, 52 insertions(+), 20 deletions(-) diff --git a/sys/dev/gve/gve_adminq.h b/sys/dev/gve/gve_adminq.h index bc51046a3037..531a844f7d90 100644 --- a/sys/dev/gve/gve_adminq.h +++ b/sys/dev/gve/gve_adminq.h @@ -377,7 +377,8 @@ struct stats { _Static_assert(sizeof(struct stats) == 16, "gve: bad admin queue struct length"); -/* These are control path types for PTYPE which are the same as the data path +/* + * These are control path types for PTYPE which are the same as the data path * types. */ struct gve_ptype_entry { diff --git a/sys/dev/gve/gve_dqo.h b/sys/dev/gve/gve_dqo.h index 214138303a77..212bfa1a6ad3 100644 --- a/sys/dev/gve/gve_dqo.h +++ b/sys/dev/gve/gve_dqo.h @@ -208,9 +208,14 @@ _Static_assert(sizeof(struct gve_tx_metadata_dqo) == 12, #define GVE_TX_METADATA_VERSION_DQO 0 +/* Used to access the generation bit within a TX completion descriptor. */ +#define GVE_TX_DESC_DQO_GEN_BYTE_OFFSET 1 +#define GVE_TX_DESC_DQO_GEN_BIT_MASK 0x80 + /* TX completion descriptor */ struct gve_tx_compl_desc_dqo { - /* For types 0-4 this is the TX queue ID associated with this + /* + * For types 0-4 this is the TX queue ID associated with this * completion. */ uint16_t id:11; @@ -222,12 +227,14 @@ struct gve_tx_compl_desc_dqo { /* Flipped by HW to notify the descriptor is populated. */ uint16_t generation:1; union { - /* For descriptor completions, this is the last index fetched + /* + * For descriptor completions, this is the last index fetched * by HW + 1. */ __le16 tx_head; - /* For packet completions, this is the completion tag set on the + /* + * For packet completions, this is the completion tag set on the * TX packet descriptors. */ __le16 completion_tag; @@ -258,6 +265,10 @@ struct gve_rx_desc_dqo { _Static_assert(sizeof(struct gve_rx_desc_dqo) == 32, "gve: bad dqo desc struct length"); +/* Used to access the generation bit within an RX completion descriptor. */ +#define GVE_RX_DESC_DQO_GEN_BYTE_OFFSET 5 +#define GVE_RX_DESC_DQO_GEN_BIT_MASK 0x40 + /* Descriptor for HW to notify SW of new packets received on RX queue. */ struct gve_rx_compl_desc_dqo { /* Must be 1 */ @@ -266,7 +277,8 @@ struct gve_rx_compl_desc_dqo { /* Packet originated from this system rather than the network. */ uint8_t loopback:1; - /* Set when IPv6 packet contains a destination options header or routing + /* + * Set when IPv6 packet contains a destination options header or routing * header. */ uint8_t ipv6_ex_add:1; diff --git a/sys/dev/gve/gve_rx_dqo.c b/sys/dev/gve/gve_rx_dqo.c index a499ac9d3c6a..11b2c7ea0c55 100644 --- a/sys/dev/gve/gve_rx_dqo.c +++ b/sys/dev/gve/gve_rx_dqo.c @@ -962,6 +962,19 @@ drop_frag_clear_ctx: rx->ctx = (struct gve_rx_ctx){}; } +static uint8_t +gve_rx_get_gen_bit(uint8_t *desc) +{ + uint8_t byte; + + /* + * Prevent generation bit from being read after the rest of the + * descriptor. + */ + byte = atomic_load_acq_8(desc + GVE_RX_DESC_DQO_GEN_BYTE_OFFSET); + return ((byte & GVE_RX_DESC_DQO_GEN_BIT_MASK) != 0); +} + static bool gve_rx_cleanup_dqo(struct gve_priv *priv, struct gve_rx_ring *rx, int budget) { @@ -971,17 +984,14 @@ gve_rx_cleanup_dqo(struct gve_priv *priv, struct gve_rx_ring *rx, int budget) NET_EPOCH_ASSERT(); while (work_done < budget) { - bus_dmamap_sync(rx->dqo.compl_ring_mem.tag, rx->dqo.compl_ring_mem.map, + bus_dmamap_sync(rx->dqo.compl_ring_mem.tag, + rx->dqo.compl_ring_mem.map, BUS_DMASYNC_POSTREAD); compl_desc = &rx->dqo.compl_ring[rx->dqo.tail]; - if (compl_desc->generation == rx->dqo.cur_gen_bit) + if (gve_rx_get_gen_bit((uint8_t *)compl_desc) == + rx->dqo.cur_gen_bit) break; - /* - * Prevent generation bit from being read after the rest of the - * descriptor. - */ - atomic_thread_fence_acq(); rx->cnt++; rx->dqo.tail = (rx->dqo.tail + 1) & rx->dqo.mask; diff --git a/sys/dev/gve/gve_tx_dqo.c b/sys/dev/gve/gve_tx_dqo.c index 7361d47b8ce6..8a1993c3e712 100644 --- a/sys/dev/gve/gve_tx_dqo.c +++ b/sys/dev/gve/gve_tx_dqo.c @@ -1029,6 +1029,19 @@ gve_clear_tx_ring_dqo(struct gve_priv *priv, int i) gve_tx_clear_compl_ring_dqo(tx); } +static uint8_t +gve_tx_get_gen_bit(uint8_t *desc) +{ + uint8_t byte; + + /* + * Prevent generation bit from being read after the rest of the + * descriptor. + */ + byte = atomic_load_acq_8(desc + GVE_TX_DESC_DQO_GEN_BYTE_OFFSET); + return ((byte & GVE_TX_DESC_DQO_GEN_BIT_MASK) != 0); +} + static bool gve_tx_cleanup_dqo(struct gve_priv *priv, struct gve_tx_ring *tx, int budget) { @@ -1041,20 +1054,16 @@ gve_tx_cleanup_dqo(struct gve_priv *priv, struct gve_tx_ring *tx, int budget) uint16_t type; while (work_done < budget) { - bus_dmamap_sync(tx->dqo.compl_ring_mem.tag, tx->dqo.compl_ring_mem.map, + bus_dmamap_sync(tx->dqo.compl_ring_mem.tag, + tx->dqo.compl_ring_mem.map, BUS_DMASYNC_POSTREAD); compl_desc = &tx->dqo.compl_ring[tx->dqo.compl_head]; - if (compl_desc->generation == tx->dqo.cur_gen_bit) + if (gve_tx_get_gen_bit((uint8_t *)compl_desc) == + tx->dqo.cur_gen_bit) break; - /* - * Prevent generation bit from being read after the rest of the - * descriptor. - */ - atomic_thread_fence_acq(); type = compl_desc->type; - if (type == GVE_COMPL_TYPE_DQO_DESC) { /* This is the last descriptor fetched by HW plus one */ tx_head = le16toh(compl_desc->tx_head);