From nobody Mon May 01 18:41:32 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9Brc52c4z48sLc; Mon, 1 May 2023 18:41:52 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9Brc4W4mz4Jdn; Mon, 1 May 2023 18:41:52 +0000 (UTC) (envelope-from dim@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682966512; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=WQJSoBUPDxqD7SODKYPTU7VQN3SnaoXEl9/nRJRuMyQ=; b=TWqANdDgeEecgngXZJvuolXFN/5RMTAsCXqXrEW59fGec9+fG2Fxf05YydoSa6wIqw0YSH vs50FwX5uaBfV3Fc6zwPRc+Yy5QxUvev8BkcWeVjpM3juUnPJoYt/aoFZJ+9jmI07SKwIK Xiz2OfNoNtizwBWk1wrgEQe1ywlQ0XPO6PITCvdMyJ5CYqg2WJD23wNGXlXZZ4XESBIxeI BTa9j0VvY3Gqzhx/p17mcKqcsXFStI1kpgmzz6nvHf719B6ZSeDF24ViB1klKq0w0oyk/9 30UWLDg+KBZWqM0nhVSXHP0lmTHhjfHFeVbsl1leIBHRojBpoXPiV1vVbCjBRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682966512; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=WQJSoBUPDxqD7SODKYPTU7VQN3SnaoXEl9/nRJRuMyQ=; b=PZwkcykz3CdXQM60BdTo/38r86IGkHRMGWqBQhoXofX13chI5Wb+aqKwrNtz9vph1USudM C5fFzlyhlNrzyvjMaHIfsTiq0ja1EsQ9n/c3qGWz6MCtV2KAxXehk5PZf/U3tta8zjEqp5 O4JicQDo0KyTA8Drc/nFUYBxUhSZwnh2swTyL5scwQTP8B4XM2ckZCE6hPm+dnakC4wSPu bRglotCIa6onHZ6euemFkKVaFThHZEGLkR90y30KC4LgARKARcKf6eU+afSPGaBm24Iswu DkTZ3Q1uCfb5XDXsuehmmILNONjdTwpqCAVbvoM1gyVeq2JqGIpIF5MwKYk4Qw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682966512; a=rsa-sha256; cv=none; b=Sfl4M4CE6JU2h5U8w4sNGf72VElVtL628KRg9e7iLqlRkP12Fvob1KkoCq4dnAdkivb8/p hnzS0Ps35Tv9SSOdv+HV2Bhe/XRAKs7A1BB2RkyCjLZFdfCd/EfIOmrk5eFA9P137WjkMO rqAmtyL6V90kpNVsWRnGIpPUuRz5VMvcNX5tQFE00rJqDcYDchzhv7qJRiRleklwc3txuq IXfGkEcD2bn9ugRiNosXgFbE4Rj1bYJuPM36zVaEpaKqG2OF66AsoEGOuJtCG0i4a+3Sdn mFrOX0iW/0MiDBfQzi9tI2NVcMhxa4Wbj8HFETIFhpRZ7A4UHyYauapNzkt30g== Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "tensor.andric.com", Issuer "R3" (verified OK)) (Authenticated sender: dim) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Q9Brc2rHdzssr; Mon, 1 May 2023 18:41:52 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtpclient.apple (longrow.home.andric.com [192.168.0.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 3FC596EBE8; Mon, 1 May 2023 20:41:51 +0200 (CEST) Content-Type: multipart/signed; boundary="Apple-Mail=_4E63B4C0-59F7-4609-A297-F6686495A611"; protocol="application/pgp-signature"; micalg=pgp-sha1 List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: git: 060699e91369 - stable/13 - Merge llvm-project release/15.x llvmorg-15.0.7-0-g8dfdcc7b7bf6 From: Dimitry Andric In-Reply-To: Date: Mon, 1 May 2023 20:41:32 +0200 Cc: "Jason A. Harmening" , Konstantin Belousov , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-branches@freebsd.org Message-Id: References: <202304092135.339LZMeJ081640@gitrepo.freebsd.org> <76DD2CB9-986B-4349-8F46-3B7BF63EB315@FreeBSD.org> To: John Baldwin X-Mailer: Apple Mail (2.3731.500.231) X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_4E63B4C0-59F7-4609-A297-F6686495A611 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 1 May 2023, at 18:14, John Baldwin wrote: >=20 > On 4/30/23 8:31 PM, Jason A. Harmening wrote: >> On Sun, Apr 30, 2023 at 07:34:45PM -0500, Jason A. Harmening wrote: >>> On Sun, Apr 30, 2023 at 06:47:13PM -0500, Jason A. Harmening wrote: >>>> On Sun, Apr 30, 2023 at 08:09:16AM +0300, Konstantin Belousov = wrote: >>>>> On Sat, Apr 29, 2023 at 02:27:50PM -0500, Jason A. Harmening = wrote: >>>>>> On Sat, Apr 29, 2023 at 08:49:28PM +0200, Dimitry Andric wrote: >>>>>>> On 29 Apr 2023, at 20:33, Jason A. Harmening = wrote: >>>>>>>>=20 >>>>>>>> On Sun, Apr 09, 2023 at 09:35:22PM +0000, Dimitry Andric wrote: >>>>>>>>> The branch stable/13 has been updated by dim: >>>>>>>>>=20 >>>>>>>>> URL: = https://cgit.FreeBSD.org/src/commit/?id=3D060699e9136975d51d3f726b9785bdba= c9a62ba6 >>>>>>>>>=20 >>>>>>>>> commit 060699e9136975d51d3f726b9785bdbac9a62ba6 >>>>>>>>> Author: Dimitry Andric >>>>>>>>> AuthorDate: 2023-01-14 16:33:24 +0000 >>>>>>>>> Commit: Dimitry Andric >>>>>>>>> CommitDate: 2023-04-09 14:54:52 +0000 >>>>>>>>>=20 >>>>>>>>> Merge llvm-project release/15.x = llvmorg-15.0.7-0-g8dfdcc7b7bf6 >>>>>>>>>=20 >>>>>>>>> This updates llvm, clang, compiler-rt, libc++, libunwind, = lld, lldb and >>>>>>>>> openmp to llvmorg-15.0.7-0-g8dfdcc7b7bf6. >>>>>>>>>=20 >>>>>>>>> PR: 265425 >>>>>>>>> MFC after: 2 weeks >>>>>>>>=20 >>>>>>>> This MFC of llvm15 appears to have completely broken the Intel = IOMMU >>>>>>>> driver on my stable/13 machine. After this series of commits, = any >>>>>>>> downstream DMA seems to produce an IOMMU translation fault, = which >>>>>>>> renders the machine completely unusable: no nvme boot disk, no = usb >>>>>>>> keyboard, etc. >>>>>>>>=20 >>>>>>>> The faults I see look something like this: >>>>>>>>=20 >>>>>>>> DMAR4: ahci0: pci0:17:5 sid 8d fault acc 0 adt 0x0 reason 0x3 = addr 26000 >>>>>>>>=20 >>>>>>>> It's a bit surprising to see a toolchain upgrade produce = breakage like >>>>>>>> this, but that's what git bisect clearly tells me. I wonder if = some of >>>>>>>> the IOMMU control structures might be defined as C bitfields = and the new >>>>>>>> compiler is emitting them differently? Also, was any breakage = like this >>>>>>>> observed when -current was upgraded to llvm15 several months = ago? >>>>>>>=20 >>>>>>> I haven't heard anything about such breakage, no. >>>>>>>=20 >>>>>>>=20 >>>>>>>> More generally, this is the second time in as many months I've = had to >>>>>>>> deal with IOMMU breakage on -stable. I can't imagine I'm the = only >>>>>>>> person who sees value in running with DMA remapping enabled; do = we need >>>>>>>> a dedicated DMAR-enabled machine in the cluster to smoke-test = changes >>>>>>>> like this? More generally, should we avoid MFCing high-risk = changes >>>>>>>> like this? >>>>>>>=20 >>>>>>> Since there were very few bug reports, it was not deemed high = risk. >>>>>>>=20 >>>>>>> In any case, it would be good to get the bottom of what is = causing the >>>>>>> problem, so is there any way you can isolate which code seems to = be >>>>>>> going "bad"? >>>>>>>=20 >>>>>>> For example, if this problem affects code in sys/dev/iommu, is = there >>>>>>> some way you can compile that part with -O1, or with an older = version >>>>>>> of clang (from ports), to see if the problem goes away? >>>>>>=20 >>>>>> I did try removing all custom make.conf settings (previously I = just had >>>>>> CPUTYPE?=3Dicelake-server), but that didn't change the behavior. >>>>>>=20 >>>>>> Before I try further build tweaks, I'd like to ask if the IOMMU = fault >>>>>> report can provide guidance here? AFAICT all the faults I'm = getting >>>>>> show "reason 0x3". If I'm reading the VT-d spec correctly, = FR=3D0x3 >>>>>> indicates an invalid context entry, in other words there's = something the >>>>>> hardware doesn't like in the way the address width or pagetable = base is >>>>>> configured for the PCIe requestor. >>>>>=20 >>>>> I would start looking at the other direction: might be, there are = still some >>>>> left shifts for int32 values with the shift count > 30, or uint32 = with the >>>>> count > 31. >>>>>=20 >>>>> Also might be useful to dump each context entry on creation, it is = kept >>>>> constant after. >>>>=20 >>>> I did look over the constants in intel_reg.h, and didn't see = anything >>>> that looked as though it would be susceptible to sign-extension or >>>> truncation bugs. In the failing case it's much easier for me to = catch >>>> the fault messages than any initialization message, so I = instrumented >>>> the fault handler to get the context entry from the dmar_ctx object >>>> using the same logic as dmar_map_ctx_entry(), and then dump out the = ctx1 >>>> and ctx2 fields. What I see are messages like: >>>>=20 >>>> ... ctx1 0x10013b001 ctx2 0x103 >>>>=20 >>>> At first glance these "look right": the P bit is set in ctx1, and = the >>>> rest of the field looks like a valid physical address. ctx2 also >>>> doesn't have any of the reserved bits set, but in all cases it does = have >>>> AW=3D3, which would indicate 57-bit AGAW. But when I boot the last >>>> working kernel, from the revision prior to the llvm15 MFC, I see = this in >>>> dmesg: >>>>=20 >>>> ahci0: dmar4 pci0:0:17:5 rid 8d domain 1 mgaw 48 agaw 48 re-mapped >>>>=20 >>>> ...all reported devices show 48-bit MGAW/AGAW, so I would expect = ctx2 to >>>> have AW=3D2. I suspect this may be the source of the fault, but = I'm not >>>> sure how it's getting configured that way, whether it's an issue = with >>>> reading the capability register or something else. >>>>=20 >>>=20 >>> I can confirm that hacking domain_set_agaw() to always use the = settings >>> from sagaw_bits[2] eliminates the faults and at least allows the = machine >>> to boot to single-user mode. >> I see what's happening now. When I added the hack to always set >> sagaw_bits[2], I noted that the passed-in MGAW was still 57, while >> unit->hw_cap had the correct value of 0x4 (=3D> 4-level paging, = 48-bit AW) >> in bits 12:8. The problem is that sagaw_bits has agaw=3D64 in its = last >> entry. This results in dmar_maxaddr2mgaw() attempting a comparison >> against 1ULL << 64 in the final iteration of its first loop. I = suspect >> the new compiler probably determines that last iteration is = meaningless >> and simply omits it from the (probably unrolled) loop. Since the = "loop" >> terminates with i < nitems(sagaw_bits), the subsequent "allow_less = ..." >> case doesn't execute and we end up erroneously selecting a 57-bit >> address width. Just commenting out that last entry in sagaw_bits = fixes >> the problem. >> So, two questions: >> 1) Does any VT-d hardware actually support 6-level paging? The ca. = 2021 >> VT-d spec I'm looking at indicates 5-level is the greatest depth >> supported, with everything above that being reserved. >> 2) I'd expect clang to try very hard to error out in a situation like >> this, but I see that sys/conf/kern.mk sets -Wno-shift-count-overflow >> among other things, and more of them were added for clang 15. This >> seems like a really bad idea, regardless of how much of a PITA it may = be >> to fix these warnings. >=20 > FWIW, I've been working on trying to re-enable some of the warnings = that > were disabled for clang 15 in main. I'll move that one higher up on = my > todo list. In this particular case, it doesn't warn about it though. I think KASAN might be a better 'catcher' for this kind of error, or a KUBSAN, if we had one... -Dimitry --Apple-Mail=_4E63B4C0-59F7-4609-A297-F6686495A611 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.2 iF0EARECAB0WIQR6tGLSzjX8bUI5T82wXqMKLiCWowUCZFAH3AAKCRCwXqMKLiCW o8k3AJ9DkrMB167K+sSNel6qHWAMhRILBwCfYE5kwUADLOtEslYT8QQ1pzM2sYI= =/I7V -----END PGP SIGNATURE----- --Apple-Mail=_4E63B4C0-59F7-4609-A297-F6686495A611--