From nobody Tue May 20 23:51:15 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4b2BCR4wFjz5wC22; Tue, 20 May 2025 23:51:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4b2BCR2J0mz3k4r; Tue, 20 May 2025 23:51:15 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1747785075; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ScWVt/IvU1F6ibs3DkfxMdWPGtEBHEkf+8a8GaQprVg=; b=CrhsH5jljgDG1kaNJjBESEICITRI8qggVQC7R3Y48fnrgsoSp8U5bhQ/yYlyu/nn4yQt3c Jl5bSav3Jx9Tp9ZKYo1Tif+VgOdex67+hJl7iG3GSKSGxN0oofUfU6WCBSlRHPL4jjxG30 qn7NwysHJ5veSZdP4Up1m9PPkzaLj9kZ0UAJepLgp9Oa0+2FhNTCAIXB5l3KyEo+Yr2DWP tGrMgURW+ak2MTF29xNHpRsAfU34z3fdABstRQyVb6TS/7VURSKV9wDBPRE9CmTBDLojrC x7Hmk6X4u6R5qmCogzfMC7ik2AfopRjn2S+dyYuLQ8GuBlNXkuOSff9le3RVww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1747785075; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ScWVt/IvU1F6ibs3DkfxMdWPGtEBHEkf+8a8GaQprVg=; b=xZA9AG7ZQLdG1UmiU27Lyv8FqN4NN5ZC6CBgm3b7nUQjFfguMX//v0r+70Tbc+Mo0Lo8ts QiwL4VW8TBFxp//WrfQ6GDgIvBpCBx4Qpe1QICwnZIJvyCNzlYluLKdpB+7zrB6r6qnVKJ UJIa1St6BYMbrSTh7WHi5oY5eR+E62uZ4h3szzsNidxhR81oF2eTluFBqaNQrkj2su4SMS TtPK8KWUxuz7WCm9wiy2rLOhRm2L/jLzoeXKGavdL60HAWz3b80DFYtuxx+OEDS44CKhCB 4gjxcYNr45uw5SSmiSe3xqHHMJKtq5M0uJjNI1O5hXxSxQ8qYX6BGO02bSLRZQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1747785075; a=rsa-sha256; cv=none; b=ZHXLZR0PlUx5Og8mjcUV+GCsCY35k9woXQvUI2gARhTVdkBJAtRin/v9AE4CbOsG0Tvxcx RlQ1QoryXWdnaAAbV1sYaaUkpWOWb0ELPgSrQY9Qhw3pw7GHPWcL3rheK4zwSvQSZkFL6Z sf9Ou3UCLsy9cjT5q+fAHFMoysPUf0fMUgDKRM1ih9mrAWwNI1lTLsND2Qmb91ilO2xEGe 6vdjyCEK+mtji6VRv7gDu+FyiX0bYUWJo9o+pX3yDYLWhD3JOyMHh+ggYXGLOiI689T+yb Buo58PbFdjnJW92OHndR1iqgG1fFoy5HFVv4jcVYCzk/h6jfOtgQRJ7ZQA7Z8w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4b2BCR1ktjz19f7; Tue, 20 May 2025 23:51:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 54KNpF2q058708; Tue, 20 May 2025 23:51:15 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 54KNpFWE058687; Tue, 20 May 2025 23:51:15 GMT (envelope-from git) Date: Tue, 20 May 2025 23:51:15 GMT Message-Id: <202505202351.54KNpFWE058687@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: b044f125373e - main - gve: Use load-acquire to fetch generation bits List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b044f125373e05c20ca16edfd29cccad1f4025bd Auto-Submitted: auto-generated The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=b044f125373e05c20ca16edfd29cccad1f4025bd commit b044f125373e05c20ca16edfd29cccad1f4025bd Author: Jasper Tran O'Leary AuthorDate: 2025-05-20 23:34:44 +0000 Commit: Mark Johnston CommitDate: 2025-05-20 23:50:59 +0000 gve: Use load-acquire to fetch generation bits When running the driver using the DQO queue format, we must load the generation bit and check it before possibly reading the rest of the descriptor's fields. Previously, we guarded against reordering of reads using an explicit thread fence. This commit changes the thread fence to a load with acquire semantics. Because the tx and rx generation fields are in a bitfield, we cannot explicitly address them in an atomic load. Instead we load the respective containing bytes in the descriptor and mask them appropriately. Signed-off-by: Jasper Tran O'Leary Reviewed by: markj MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D50384 --- sys/dev/gve/gve_adminq.h | 3 ++- sys/dev/gve/gve_dqo.h | 20 ++++++++++++++++---- sys/dev/gve/gve_rx_dqo.c | 24 +++++++++++++++++------- sys/dev/gve/gve_tx_dqo.c | 25 +++++++++++++++++-------- 4 files changed, 52 insertions(+), 20 deletions(-) diff --git a/sys/dev/gve/gve_adminq.h b/sys/dev/gve/gve_adminq.h index bc51046a3037..531a844f7d90 100644 --- a/sys/dev/gve/gve_adminq.h +++ b/sys/dev/gve/gve_adminq.h @@ -377,7 +377,8 @@ struct stats { _Static_assert(sizeof(struct stats) == 16, "gve: bad admin queue struct length"); -/* These are control path types for PTYPE which are the same as the data path +/* + * These are control path types for PTYPE which are the same as the data path * types. */ struct gve_ptype_entry { diff --git a/sys/dev/gve/gve_dqo.h b/sys/dev/gve/gve_dqo.h index 214138303a77..212bfa1a6ad3 100644 --- a/sys/dev/gve/gve_dqo.h +++ b/sys/dev/gve/gve_dqo.h @@ -208,9 +208,14 @@ _Static_assert(sizeof(struct gve_tx_metadata_dqo) == 12, #define GVE_TX_METADATA_VERSION_DQO 0 +/* Used to access the generation bit within a TX completion descriptor. */ +#define GVE_TX_DESC_DQO_GEN_BYTE_OFFSET 1 +#define GVE_TX_DESC_DQO_GEN_BIT_MASK 0x80 + /* TX completion descriptor */ struct gve_tx_compl_desc_dqo { - /* For types 0-4 this is the TX queue ID associated with this + /* + * For types 0-4 this is the TX queue ID associated with this * completion. */ uint16_t id:11; @@ -222,12 +227,14 @@ struct gve_tx_compl_desc_dqo { /* Flipped by HW to notify the descriptor is populated. */ uint16_t generation:1; union { - /* For descriptor completions, this is the last index fetched + /* + * For descriptor completions, this is the last index fetched * by HW + 1. */ __le16 tx_head; - /* For packet completions, this is the completion tag set on the + /* + * For packet completions, this is the completion tag set on the * TX packet descriptors. */ __le16 completion_tag; @@ -258,6 +265,10 @@ struct gve_rx_desc_dqo { _Static_assert(sizeof(struct gve_rx_desc_dqo) == 32, "gve: bad dqo desc struct length"); +/* Used to access the generation bit within an RX completion descriptor. */ +#define GVE_RX_DESC_DQO_GEN_BYTE_OFFSET 5 +#define GVE_RX_DESC_DQO_GEN_BIT_MASK 0x40 + /* Descriptor for HW to notify SW of new packets received on RX queue. */ struct gve_rx_compl_desc_dqo { /* Must be 1 */ @@ -266,7 +277,8 @@ struct gve_rx_compl_desc_dqo { /* Packet originated from this system rather than the network. */ uint8_t loopback:1; - /* Set when IPv6 packet contains a destination options header or routing + /* + * Set when IPv6 packet contains a destination options header or routing * header. */ uint8_t ipv6_ex_add:1; diff --git a/sys/dev/gve/gve_rx_dqo.c b/sys/dev/gve/gve_rx_dqo.c index a499ac9d3c6a..11b2c7ea0c55 100644 --- a/sys/dev/gve/gve_rx_dqo.c +++ b/sys/dev/gve/gve_rx_dqo.c @@ -962,6 +962,19 @@ drop_frag_clear_ctx: rx->ctx = (struct gve_rx_ctx){}; } +static uint8_t +gve_rx_get_gen_bit(uint8_t *desc) +{ + uint8_t byte; + + /* + * Prevent generation bit from being read after the rest of the + * descriptor. + */ + byte = atomic_load_acq_8(desc + GVE_RX_DESC_DQO_GEN_BYTE_OFFSET); + return ((byte & GVE_RX_DESC_DQO_GEN_BIT_MASK) != 0); +} + static bool gve_rx_cleanup_dqo(struct gve_priv *priv, struct gve_rx_ring *rx, int budget) { @@ -971,17 +984,14 @@ gve_rx_cleanup_dqo(struct gve_priv *priv, struct gve_rx_ring *rx, int budget) NET_EPOCH_ASSERT(); while (work_done < budget) { - bus_dmamap_sync(rx->dqo.compl_ring_mem.tag, rx->dqo.compl_ring_mem.map, + bus_dmamap_sync(rx->dqo.compl_ring_mem.tag, + rx->dqo.compl_ring_mem.map, BUS_DMASYNC_POSTREAD); compl_desc = &rx->dqo.compl_ring[rx->dqo.tail]; - if (compl_desc->generation == rx->dqo.cur_gen_bit) + if (gve_rx_get_gen_bit((uint8_t *)compl_desc) == + rx->dqo.cur_gen_bit) break; - /* - * Prevent generation bit from being read after the rest of the - * descriptor. - */ - atomic_thread_fence_acq(); rx->cnt++; rx->dqo.tail = (rx->dqo.tail + 1) & rx->dqo.mask; diff --git a/sys/dev/gve/gve_tx_dqo.c b/sys/dev/gve/gve_tx_dqo.c index 7361d47b8ce6..8a1993c3e712 100644 --- a/sys/dev/gve/gve_tx_dqo.c +++ b/sys/dev/gve/gve_tx_dqo.c @@ -1029,6 +1029,19 @@ gve_clear_tx_ring_dqo(struct gve_priv *priv, int i) gve_tx_clear_compl_ring_dqo(tx); } +static uint8_t +gve_tx_get_gen_bit(uint8_t *desc) +{ + uint8_t byte; + + /* + * Prevent generation bit from being read after the rest of the + * descriptor. + */ + byte = atomic_load_acq_8(desc + GVE_TX_DESC_DQO_GEN_BYTE_OFFSET); + return ((byte & GVE_TX_DESC_DQO_GEN_BIT_MASK) != 0); +} + static bool gve_tx_cleanup_dqo(struct gve_priv *priv, struct gve_tx_ring *tx, int budget) { @@ -1041,20 +1054,16 @@ gve_tx_cleanup_dqo(struct gve_priv *priv, struct gve_tx_ring *tx, int budget) uint16_t type; while (work_done < budget) { - bus_dmamap_sync(tx->dqo.compl_ring_mem.tag, tx->dqo.compl_ring_mem.map, + bus_dmamap_sync(tx->dqo.compl_ring_mem.tag, + tx->dqo.compl_ring_mem.map, BUS_DMASYNC_POSTREAD); compl_desc = &tx->dqo.compl_ring[tx->dqo.compl_head]; - if (compl_desc->generation == tx->dqo.cur_gen_bit) + if (gve_tx_get_gen_bit((uint8_t *)compl_desc) == + tx->dqo.cur_gen_bit) break; - /* - * Prevent generation bit from being read after the rest of the - * descriptor. - */ - atomic_thread_fence_acq(); type = compl_desc->type; - if (type == GVE_COMPL_TYPE_DQO_DESC) { /* This is the last descriptor fetched by HW plus one */ tx_head = le16toh(compl_desc->tx_head);