From nobody Wed Nov 18 19:24:58 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4jVBPk2T1dz715jF for ; Wed, 18 Nov 2026 19:24:58 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4jVBPk1fLcz3F1n for ; Wed, 18 Nov 2026 19:24:58 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1795029898; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ZmwwpZY68VrOQolQQi3OGmLf5pDq00fXiEtcU0a3HzM=; b=XwqO/bCL7NfCuZUOBpWGmWfUc3TRFhPGZF8IB9JebvQ/6ESiYDx4hERnYBWrnhlcApYS+g rz+wvXBpyiP/XNCv2hq5TirUvPSU49mouoNU4ig8qU6yAF017IE6FmPEaIvOrvE4BjFtnx sOQbDi0PtNPVzffZAOxlSFcdH/rNzH7NlavdDj+u9WlMB3GRg9NHGIyrBBG7DMgTeT7UVZ uvXwpBOWZYQ36oUInvLD9cfIQA2AefBxPaexOSQ75a4qsdqEE7qHT++4OucHeNNZQEu7uc 1WT9cBH+iMRALOagnbhMo7OFGMZ0yhl9plGvT6yfjtI5Sw10k91nSDq4s69eFQ== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1795029898; b=Qn49tcFdN/yiWh3vEKmt+5POtwt2I0GOfjgS1i6TKZ5jOAuMIuFwRkonDco08WtqN1PVAI yBoWHIx+UoQX0CA/qZjxsUrxzTQKlSm9eNQPib4Jgg7zo+3X4Ti4JfGouQczTy84oZ9GNJ CksLVQXVd1Bo3UeJ9ajxSRyg+Q4Lx/9WGlJjPQSWEpDxJkCsjEGvaQFkLDbRqH6sl3CllW 9ZS9F+Y80X2Yrqivq6AYgQs7V0S002hExTG9qjQ9Dv+YvmvPHr927essRxwgxOmtibelwf xf0ZvDg83rPg9KJQjMFubWLugMPLEfq9oK/es/cu/pj2y9hHWrbJ8E2nK+yTxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1795029898; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ZmwwpZY68VrOQolQQi3OGmLf5pDq00fXiEtcU0a3HzM=; b=pvi5GtnP4QzfW1RirZREqmZQsFOi1OumMSdY3xNzns+H7K0DUj046o6yCAPRbXGvDycuz4 tMqgjPPnT1UR5MaJHNfBRW+OdZHxvcsPHFMna1/hxjOnknOyi8nsUsjrJg5IYnZL9XmI6x /dmSbNkUEoZn9FcuYAW4J1eTJOxNfbAjQ5ZbH5YFpAPHDrxNIF7McUto0tatoYXp/H3siG O5q241ZnVTTDhHq8fdLz3v9Vv9fhlSjhEV/F2vtAC+acIipIepDbjVMDs3j7Zo23zQTAE8 DygsHn79TWAWdu/pUO4oOD36t9jexZUmPvVFBmz7TxBQtWRsOC092tOyQRyVUg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4jVBPk1JPkzRGR for ; Wed, 18 Nov 2026 19:24:58 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 16ac3 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Wed, 18 Nov 2026 19:24:58 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Cc: Tim Hostetler From: Mark Johnston Subject: git: 62866fce4dc9 - main - gve: Fix DQO TX free-list memory ordering and pending_pkt state List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all X-BeenThere: dev-commits-src-all@freebsd.org List-Id: Commit messages for all branches of the src repository List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 62866fce4dc9c6595ef8c0b5b003f52ebf10067e Auto-Submitted: auto-generated Date: Wed, 18 Nov 2026 19:24:58 +0000 Message-Id: <6afdfb8a.16ac3.42d90085@gitrepo.freebsd.org> The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=62866fce4dc9c6595ef8c0b5b003f52ebf10067e commit 62866fce4dc9c6595ef8c0b5b003f52ebf10067e Author: Tim Hostetler AuthorDate: 2026-11-18 14:23:30 +0000 Commit: Mark Johnston CommitDate: 2026-11-18 19:24:35 +0000 gve: Fix DQO TX free-list memory ordering and pending_pkt state The DQO TX path uses lockless producer/consumer free lists (free_pending_pkts_prd/csm and free_qpl_bufs_prd/csm) shared between the completion and transmit taskqueues. On weakly ordered architectures such as arm64, the current synchronization has three issues: 1. gve_free_pending_packet() updates pending_pkt state and links pending_pkt->next before pushing the entry onto free_pending_pkts_prd using a relaxed atomic_cmpset_32(), while gve_alloc_pending_packet() steals free_pending_pkts_prd using a relaxed atomic_swap_32(). Without release/acquire ordering, the transmit thread can observe a stale pending_pkt->next pointer after stealing the producer list (either orphaning freed entries or re-allocating entries whose completion tags are still in flight), or prior completion stores (such as setting state to FREE or clearing mbuf) can become visible after the slot is re-allocated. 2. Similarly, in QPL mode, gve_tx_alloc_qpl_buf() steals free_qpl_bufs_prd with a relaxed atomic_swap_32() without an acquire fence to pair with atomic_cmpset_rel_32() in gve_reap_qpl_bufs_dqo(). 3. gve_alloc_pending_packet() marks pending_pkt->state as GVE_PACKET_STATE_PENDING_DATA_COMPL and sets the enqueue timestamp before pkt->mbuf is populated and before fallible checks in the transmit path succeed. If a completion arrives for a duplicate tag (from issue 1) while the second transmit is still in progress or after it aborts, gve_handle_packet_completion() observes a pending state while pkt->mbuf is NULL. This was discovered during multi-stream TCP testing on ARM64 GCP instances running FreeBSD 14.3-RELEASE, where it caused spurious missing-completion logs followed by a NULL pointer dereference of pending_pkt->mbuf (offset 0x38) in gve_handle_packet_completion(): gve0: No pending data completion: 0 gve0: No pending data completion: 1 gve0: No pending data completion: 2 Fatal data abort: x0: 0xffffe00009d61e00 x1: 0xffffe00009ce6580 ... elr: 0xffff0000013b49d8 (gve_tx_cleanup_tq_dqo + 0x120) far: 0x0000000000000038 panic: vm_fault failed: 0xffff0000013b49d8 error 1 cpuid = 14 KDB: stack backtrace: #0 0xffff000000516174 at kdb_backtrace+0x60 #1 0xffff0000004bc4c0 at vpanic+0x13c #2 0xffff0000004bc380 at panic+0x44 #3 0xffff0000008d22bc at data_abort+0x314 #4 0xffff0000008b1878 at handle_el1h_sync+0x18 #5 0xffff0000013b49d8 at gve_tx_cleanup_tq_dqo+0x120 #6 0xffff00000052d4f0 at taskqueue_run_locked+0x17c #7 0xffff00000052e4f0 at taskqueue_thread_loop+0xc8 Use atomic_cmpset_rel_32() when pushing onto free_pending_pkts_prd, add atomic_thread_fence_acq() after stealing free_pending_pkts_prd in gve_alloc_pending_packet() and free_qpl_bufs_prd in gve_tx_alloc_qpl_buf(), clear pending_pkt->next upon allocation, and defer populating pkt->mbuf, pkt->state, and the enqueue timestamp until all fallible checks in the transmit path have succeeded. Additionally, replace atomic_load_acq_32() with a relaxed atomic_load_32() when reading old_head in gve_free_pending_packet(), as old_head is only used as a scalar index and never dereferenced by the producer. Signed-off-by: Tim Hostetler Fixes: d438b4ef0cfc ("gve: Add DQO RDA support") Fixes: 2348ac893d10 ("gve: Add DQO QPL support") Reviewed by: markj, Jasper Tran O'Leary Differential Revision: https://reviews.freebsd.org/D61233 --- sys/dev/gve/gve_tx_dqo.c | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/sys/dev/gve/gve_tx_dqo.c b/sys/dev/gve/gve_tx_dqo.c index e2dcedb43bbb..cac82a7ca75e 100644 --- a/sys/dev/gve/gve_tx_dqo.c +++ b/sys/dev/gve/gve_tx_dqo.c @@ -515,6 +515,13 @@ gve_alloc_pending_packet(struct gve_tx_ring *tx) if (__predict_false(index == -1)) { tx->dqo.free_pending_pkts_csm = atomic_swap_32( &tx->dqo.free_pending_pkts_prd, -1); + /* + * Pairs with atomic_cmpset_rel_32() in + * gve_free_pending_packet() so the completion thread's writes + * to all slots in the stolen list are visible before this xmit + * thread walks ->next and reuses them. + */ + atomic_thread_fence_acq(); index = tx->dqo.free_pending_pkts_csm; if (__predict_false(index == -1)) @@ -525,9 +532,7 @@ gve_alloc_pending_packet(struct gve_tx_ring *tx) /* Remove pending_pkt from the consumer list */ tx->dqo.free_pending_pkts_csm = pending_pkt->next; - pending_pkt->state = GVE_PACKET_STATE_PENDING_DATA_COMPL; - - gve_set_timestamp(&pending_pkt->enqueue_time_sec); + pending_pkt->next = -1; return (pending_pkt); } @@ -545,10 +550,16 @@ gve_free_pending_packet(struct gve_tx_ring *tx, /* Add pending_pkt to the producer list */ while (true) { - old_head = atomic_load_acq_32(&tx->dqo.free_pending_pkts_prd); + old_head = atomic_load_32(&tx->dqo.free_pending_pkts_prd); pending_pkt->next = old_head; - if (atomic_cmpset_32(&tx->dqo.free_pending_pkts_prd, + /* + * Pairs with atomic_thread_fence_acq() in + * gve_alloc_pending_packet() so this completion thread's + * updates to state and ->next are visible before the xmit + * thread steals and reuses this slot. + */ + if (atomic_cmpset_rel_32(&tx->dqo.free_pending_pkts_prd, old_head, index)) break; } @@ -621,6 +632,14 @@ gve_tx_alloc_qpl_buf(struct gve_tx_ring *tx) if (__predict_false(buf == -1)) { tx->dqo.free_qpl_bufs_csm = atomic_swap_32( &tx->dqo.free_qpl_bufs_prd, -1); + /* + * Pairs with atomic_cmpset_rel_32() in + * gve_reap_qpl_bufs_dqo() so the completion thread's + * qpl_bufs[] entries for the entire stolen list are visible + * before this xmit thread walks and allocates them. + */ + atomic_thread_fence_acq(); + buf = tx->dqo.free_qpl_bufs_csm; if (__predict_false(buf == -1)) return (-1); @@ -748,12 +767,15 @@ gve_xmit_dqo_qpl(struct gve_tx_ring *tx, struct mbuf *mbuf) return (ENOBUFS); } completion_tag = pkt - tx->dqo.pending_pkts; - pkt->mbuf = mbuf; err = gve_tx_fill_ctx_descs(tx, mbuf, is_tso, &desc_idx); if (err) goto abort; + pkt->mbuf = mbuf; + pkt->state = GVE_PACKET_STATE_PENDING_DATA_COMPL; + gve_set_timestamp(&pkt->enqueue_time_sec); + gve_tx_copy_mbuf_and_write_pkt_descs(tx, mbuf, pkt, has_csum_flag, completion_tag, &desc_idx); @@ -770,7 +792,6 @@ gve_xmit_dqo_qpl(struct gve_tx_ring *tx, struct mbuf *mbuf) return (0); abort: - pkt->mbuf = NULL; gve_free_pending_packet(tx, pkt); return (err); } @@ -823,7 +844,6 @@ gve_xmit_dqo(struct gve_tx_ring *tx, struct mbuf **mbuf_ptr) if (err) goto abort; mbuf = *mbuf_ptr; /* gve_map_mbuf_dqo might replace the mbuf chain */ - pkt->mbuf = mbuf; total_descs_needed = 1 + /* general_ctx_desc */ nsegs + /* pkt_desc */ @@ -838,6 +858,10 @@ gve_xmit_dqo(struct gve_tx_ring *tx, struct mbuf **mbuf_ptr) if (err) goto abort_with_dma; + pkt->mbuf = mbuf; + pkt->state = GVE_PACKET_STATE_PENDING_DATA_COMPL; + gve_set_timestamp(&pkt->enqueue_time_sec); + bus_dmamap_sync(tx->dqo.buf_dmatag, pkt->dmamap, BUS_DMASYNC_PREWRITE); for (i = 0; i < nsegs; i++) { gve_tx_fill_pkt_desc_dqo(tx, &desc_idx, @@ -861,7 +885,6 @@ gve_xmit_dqo(struct gve_tx_ring *tx, struct mbuf **mbuf_ptr) abort_with_dma: gve_unmap_packet(tx, pkt); abort: - pkt->mbuf = NULL; gve_free_pending_packet(tx, pkt); return (err); }