From nobody Wed Dec 20 15:42:53 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SwHrY3Qx5z555BK; Wed, 20 Dec 2023 15:42:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SwHrY2SfQz4gHQ; Wed, 20 Dec 2023 15:42:53 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703086973; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=IjTFNi6Kc58dNlWmJ+sI6QiHca8s/GimaQsd5Jc7lwc=; b=dV9S0ZOToaYB8FzjfsICdguogQ/4+FQaxcoifwb/CBXiuq/A1PDLg5TheduZjG5De3llxm rYPCnzpFlc5W75Algsfg3km5wIneYA2LtyODVcaM4nwKSLWWw0QbORgeu2GWlNISFii9uA N3Nqfl35KtFYX68Xwy3BLU/JAKEoNz5POVesOhS1CeG4AvU59cARQb1jr0T0jJlacXn6ag l4KjQdgBnvBDW08kRoV+dRC/PAxvwGam6Npb3kSvpmYkamPTWkyNvdTn4Q7Lkp18ik6Aqa wJd8bLQ4OuVPNGwucgKGW03KvNwxgWc2Rn4E4j1Ygwdt1e9XcwR1x6GozHODiQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703086973; a=rsa-sha256; cv=none; b=sK4wuHULPQK2l773SefQX6SaQEq51JoX7Z8l1nxP0qGjtdyzuNwO+Ct4zBnqJaUjwpEMzl Us8Q5muFw1LZ6Rt4b8j+1nh4xAMV9pjYCMV7SM/WT6HscF/lPFBJEnyVBwZ9a5AoTcXFk7 dh32MXbolx34I/3YBQsn6RxVD0Ys2c+QcLdcP9GjISeDa5w9JQfKwwTYMe4QzL6AVAXa+Y d7JKuIre9a1C5C7PgFVFpUoE+l3tu4P/lpijVeR+CFPnPBv6TEQvU4pV7GcEzz1h/QDzPl VY3b62iY1hUFf2CWcYbwUxS/jT6DfbcnUneZYN0Qpb4GBUs0tICo3uCQ9DdS9w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703086973; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=IjTFNi6Kc58dNlWmJ+sI6QiHca8s/GimaQsd5Jc7lwc=; b=d9fFyKyg7hinSIj0M0DxYFihmEdpp8E5ncrN/QBOsoACrbkJ6H2scu/qD5GyxK0cM++xuR YOd1gjX04BsnHvMwrDK7uI3yIdK4TXixOoGzpKQ9zTG7d/OZjfdUSQNKENwRaPA2wPyUsk 7dOm6aRzTFtnn7V7ZaDzPEw0iOCsXGBUlGLW7D2fk2FiAq4B1vsI9pvBgIR4n/nKkN1iWJ rJ67qMXM+2tBgNkOxGT+ZO5M+9TnSylmZca02hz74Gm7+WyhAn9ZzdAFz4JrK6Q9d+wtw4 gvAsxoQBda1tvCiEBeDeAi6NIwAvFcXxjSwocUL9lFnQB1MRj5ldnzPpaYZn0Q== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SwHrY1Wfmztq1; Wed, 20 Dec 2023 15:42:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3BKFgrEO064026; Wed, 20 Dec 2023 15:42:53 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3BKFgrHn064023; Wed, 20 Dec 2023 15:42:53 GMT (envelope-from git) Date: Wed, 20 Dec 2023 15:42:53 GMT Message-Id: <202312201542.3BKFgrHn064023@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: "Kenneth D. Merry" Subject: git: 504e85ade103 - stable/14 - mpr, mps: Establish busdma boundaries for memory pools List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ken X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 504e85ade103b0c2cafefb2d9dea86e94aef779f Auto-Submitted: auto-generated The branch stable/14 has been updated by ken: URL: https://cgit.FreeBSD.org/src/commit/?id=504e85ade103b0c2cafefb2d9dea86e94aef779f commit 504e85ade103b0c2cafefb2d9dea86e94aef779f Author: Kenneth D. Merry AuthorDate: 2023-12-14 20:05:17 +0000 Commit: Kenneth D. Merry CommitDate: 2023-12-20 15:40:42 +0000 mpr, mps: Establish busdma boundaries for memory pools Most all of the memory used by the cards in the mpr(4) and mps(4) drivers is required, according to the specs and Broadcom developers, to be within a 4GB segment of memory. This includes: System Request Message Frames pool Reply Free Queues pool ReplyDescriptorPost Queues pool Chain Segments pool Sense Buffers pool SystemReply message pool We got a bug report from Dwight Engen, who ran into data corruption in the BAE port of FreeBSD: > We have a port of the FreeBSD mpr driver to our kernel and recently > I found an issue under heavy load where a DMA may go to the wrong > address. The test system is a Supermicro X10SRH-CLN4F with the > onboard SAS3008 controller setup with 2 enterprise Micron SSDs in > RAID 0 (striped). I have debugged the issue and narrowed down that > the errant DMA is one that has a segment that crosses a 4GB > physical boundary. There are more details I can provide if you'd > like, but with the attached patch in place I can no longer > re-create the issue. > I'm not sure if this is a known limit of the card (have not found a > datasheet/programming docs for the chip) or our system is just > doing something a bit different. Any helpful info or insight would > be welcome. > Anyway, just thought this might be helpful info if you want to > apply a similar fix to FreeBSD. You can ignore/discard the commit > message as it is my internal commit (blkio is our own tool we use > to write/read every block of a device with CRC verification which > is how I found the problem). The commit message was: > [PATCH 8/9] mpr: fix memory corrupting DMA when sg segment crosses > 4GB boundary > Test case was two SSD's in RAID 0 (stripe). The logical disk was > then partitioned into two partitions. One partition had lots of > filesystem I/O and the other was initially filled using blkio with > CRCable data and then read back with blkio CRC verify in a loop. > Eventually blkio would report a bad CRC block because the physical > page being read-ahead into didn't contain the right data. If the > physical address in the arq/segs was for example 0x500003000 the > data would actually be DMAed to 0x400003000. The original patch was against mpr(4) before busdma templates were introduced, and only affected the buffer pool (sc->buffer_dmat) in the mpr(4) driver. After some discussion with Dwight and the LSI/Broadcom developers and looking through the driver, it looks like most of the queues in the driver are ok, because they limit the memory used to memory below 4GB. The buffer queue and the chain frames seem to be the exceptions. This is pretty much the same between the mpr(4) and mps(4) drivers. So, apply a 4GB boundary limitation for the buffer and chain frame pools in the mpr(4) and mps(4) drivers. Reported by: Dwight Engen Reviewed by: imp Obtained from: Dwight Engen Differential Revision: (cherry picked from commit 264610a86e14f8e123d94c3c3bd9632d75c078a3) --- sys/dev/mpr/mpr.c | 6 ++++-- sys/dev/mps/mps.c | 6 ++++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/sys/dev/mpr/mpr.c b/sys/dev/mpr/mpr.c index 23f7ff0c3e9a..d5c02f9608ca 100644 --- a/sys/dev/mpr/mpr.c +++ b/sys/dev/mpr/mpr.c @@ -1500,7 +1500,8 @@ mpr_alloc_requests(struct mpr_softc *sc) rsize = sc->chain_frame_size * sc->num_chains; bus_dma_template_init(&t, sc->mpr_parent_dmat); BUS_DMA_TEMPLATE_FILL(&t, BD_ALIGNMENT(16), BD_MAXSIZE(rsize), - BD_MAXSEGSIZE(rsize), BD_NSEGMENTS((howmany(rsize, PAGE_SIZE)))); + BD_MAXSEGSIZE(rsize), BD_NSEGMENTS((howmany(rsize, PAGE_SIZE))), + BD_BOUNDARY(BUS_SPACE_MAXSIZE_32BIT+1)); if (bus_dma_template_tag(&t, &sc->chain_dmat)) { mpr_dprint(sc, MPR_ERROR, "Cannot allocate chain DMA tag\n"); return (ENOMEM); @@ -1552,7 +1553,8 @@ mpr_alloc_requests(struct mpr_softc *sc) BUS_DMA_TEMPLATE_FILL(&t, BD_MAXSIZE(BUS_SPACE_MAXSIZE_32BIT), BD_NSEGMENTS(nsegs), BD_MAXSEGSIZE(BUS_SPACE_MAXSIZE_32BIT), BD_FLAGS(BUS_DMA_ALLOCNOW), BD_LOCKFUNC(busdma_lock_mutex), - BD_LOCKFUNCARG(&sc->mpr_mtx)); + BD_LOCKFUNCARG(&sc->mpr_mtx), + BD_BOUNDARY(BUS_SPACE_MAXSIZE_32BIT+1)); if (bus_dma_template_tag(&t, &sc->buffer_dmat)) { mpr_dprint(sc, MPR_ERROR, "Cannot allocate buffer DMA tag\n"); return (ENOMEM); diff --git a/sys/dev/mps/mps.c b/sys/dev/mps/mps.c index f358ab8a73a9..adad2450a3d4 100644 --- a/sys/dev/mps/mps.c +++ b/sys/dev/mps/mps.c @@ -1431,7 +1431,8 @@ mps_alloc_requests(struct mps_softc *sc) rsize = sc->reqframesz * sc->num_chains; bus_dma_template_clone(&t, sc->req_dmat); BUS_DMA_TEMPLATE_FILL(&t, BD_MAXSIZE(rsize), BD_MAXSEGSIZE(rsize), - BD_NSEGMENTS(howmany(rsize, PAGE_SIZE))); + BD_NSEGMENTS(howmany(rsize, PAGE_SIZE)), + BD_BOUNDARY(BUS_SPACE_MAXSIZE_32BIT+1)); if (bus_dma_template_tag(&t, &sc->chain_dmat)) { mps_dprint(sc, MPS_ERROR, "Cannot allocate chain DMA tag\n"); return (ENOMEM); @@ -1473,7 +1474,8 @@ mps_alloc_requests(struct mps_softc *sc) BUS_DMA_TEMPLATE_FILL(&t, BD_MAXSIZE(BUS_SPACE_MAXSIZE_32BIT), BD_NSEGMENTS(nsegs), BD_MAXSEGSIZE(BUS_SPACE_MAXSIZE_24BIT), BD_FLAGS(BUS_DMA_ALLOCNOW), BD_LOCKFUNC(busdma_lock_mutex), - BD_LOCKFUNCARG(&sc->mps_mtx)); + BD_LOCKFUNCARG(&sc->mps_mtx), + BD_BOUNDARY(BUS_SPACE_MAXSIZE_32BIT+1)); if (bus_dma_template_tag(&t, &sc->buffer_dmat)) { mps_dprint(sc, MPS_ERROR, "Cannot allocate buffer DMA tag\n"); return (ENOMEM);