git: ac64294143ff - main - arm64: Relax memory barriers around DMA operations

The branch main has been updated by andrew:

URL: https://cgit.FreeBSD.org/src/commit/?id=ac64294143ffea63c7dedf9a1859f88342bd998a

commit ac64294143ffea63c7dedf9a1859f88342bd998a
Author:     Andrew Turner <andrew@FreeBSD.org>
AuthorDate: 2026-10-13 15:36:16 +0000
Commit:     Andrew Turner <andrew@FreeBSD.org>
CommitDate: 2026-10-13 15:44:34 +0000

    arm64: Relax memory barriers around DMA operations
    
    Before and after DMA operations we need memory barriers to ensure
    stores on the CPU are observed by the device, and stores by the device
    are observed by the CPU.
    
    Previously we used a "dsb sy" instruction, however this is too strong
    and can lead to slower performance.
    
    Drew Gallatin found and tested the issue with cache-coherent devices,
    reducing the barriers to dmb instructions in the outer-shareable
    domain. After discussing the issue with the Arm Linux DMA maintainer I
    concluded it was safe for non-coherent devices as the only dsb
    instructions we need are after cache operations and these already exist.
    
    Replace the dsb before DMA with a "dmb oshst" when cache-coherent, or
    the existing dsb after invalidating the cache when not coherent. This
    ensures earlier stores have completed before we signal the device.
    
    Replace the dsb after DMA with "dmb oshld" in both cases. This ensures
    later loads or stores are ordered after earlier device register loads.
    
    Co-Authored-By: Andrew Gallatin <gallatin@FreeBSD.org>
    Sponsored by:   Arm Ltd
    Sponsored by:   Netflix
    Differential Revision:  https://reviews.freebsd.org/D60407
---
 sys/arm64/arm64/busdma_bounce.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/sys/arm64/arm64/busdma_bounce.c b/sys/arm64/arm64/busdma_bounce.c
index 49587b5f4b39..d71d114ca60a 100644
--- a/sys/arm64/arm64/busdma_bounce.c
+++ b/sys/arm64/arm64/busdma_bounce.c
@@ -1101,7 +1101,7 @@ bounce_bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map,
 		/*
 		 * Wait for any DMA operations to complete before the bcopy.
 		 */
-		dsb(sy);
+		dmb(oshld);
 	}
 	if (map == NULL)
 		goto out;
@@ -1176,11 +1176,17 @@ bounce_bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map,
 	}
 
 out:
-	if ((op & (BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE)) != 0) {
+	/*
+	 * If the device is cache-coherent then the cache operation will
+	 * have performed a dsb instruction so we don't need another
+	 * barrier here.
+	 */
+	if ((map == NULL || (map->flags & DMAMAP_COHERENT) != 0) &&
+	    (op & (BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE)) != 0) {
 		/*
 		 * Wait for the bcopy to complete before any DMA operations.
 		 */
-		dsb(sy);
+		dmb(oshst);
 	}
 
 	if (map != NULL)