From nobody Sun Apr 30 05:09:16 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8Dsk51TVz48lyG; Sun, 30 Apr 2023 05:09:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8Dsj6Mjqz3F82; Sun, 30 Apr 2023 05:09:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 33U59Gs0053410 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 30 Apr 2023 08:09:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 33U59Gs0053410 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 33U59GSK053409; Sun, 30 Apr 2023 08:09:16 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 30 Apr 2023 08:09:16 +0300 From: Konstantin Belousov To: "Jason A. Harmening" Cc: Dimitry Andric , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-branches@freebsd.org Subject: Re: git: 060699e91369 - stable/13 - Merge llvm-project release/15.x llvmorg-15.0.7-0-g8dfdcc7b7bf6 Message-ID: References: <202304092135.339LZMeJ081640@gitrepo.freebsd.org> <76DD2CB9-986B-4349-8F46-3B7BF63EB315@FreeBSD.org> List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4Q8Dsj6Mjqz3F82 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sat, Apr 29, 2023 at 02:27:50PM -0500, Jason A. Harmening wrote: > On Sat, Apr 29, 2023 at 08:49:28PM +0200, Dimitry Andric wrote: > > On 29 Apr 2023, at 20:33, Jason A. Harmening wrote: > > > > > > On Sun, Apr 09, 2023 at 09:35:22PM +0000, Dimitry Andric wrote: > > >> The branch stable/13 has been updated by dim: > > >> > > >> URL: https://cgit.FreeBSD.org/src/commit/?id=060699e9136975d51d3f726b9785bdbac9a62ba6 > > >> > > >> commit 060699e9136975d51d3f726b9785bdbac9a62ba6 > > >> Author: Dimitry Andric > > >> AuthorDate: 2023-01-14 16:33:24 +0000 > > >> Commit: Dimitry Andric > > >> CommitDate: 2023-04-09 14:54:52 +0000 > > >> > > >> Merge llvm-project release/15.x llvmorg-15.0.7-0-g8dfdcc7b7bf6 > > >> > > >> This updates llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and > > >> openmp to llvmorg-15.0.7-0-g8dfdcc7b7bf6. > > >> > > >> PR: 265425 > > >> MFC after: 2 weeks > > > > > > This MFC of llvm15 appears to have completely broken the Intel IOMMU > > > driver on my stable/13 machine. After this series of commits, any > > > downstream DMA seems to produce an IOMMU translation fault, which > > > renders the machine completely unusable: no nvme boot disk, no usb > > > keyboard, etc. > > > > > > The faults I see look something like this: > > > > > > DMAR4: ahci0: pci0:17:5 sid 8d fault acc 0 adt 0x0 reason 0x3 addr 26000 > > > > > > It's a bit surprising to see a toolchain upgrade produce breakage like > > > this, but that's what git bisect clearly tells me. I wonder if some of > > > the IOMMU control structures might be defined as C bitfields and the new > > > compiler is emitting them differently? Also, was any breakage like this > > > observed when -current was upgraded to llvm15 several months ago? > > > > I haven't heard anything about such breakage, no. > > > > > > > More generally, this is the second time in as many months I've had to > > > deal with IOMMU breakage on -stable. I can't imagine I'm the only > > > person who sees value in running with DMA remapping enabled; do we need > > > a dedicated DMAR-enabled machine in the cluster to smoke-test changes > > > like this? More generally, should we avoid MFCing high-risk changes > > > like this? > > > > Since there were very few bug reports, it was not deemed high risk. > > > > In any case, it would be good to get the bottom of what is causing the > > problem, so is there any way you can isolate which code seems to be > > going "bad"? > > > > For example, if this problem affects code in sys/dev/iommu, is there > > some way you can compile that part with -O1, or with an older version > > of clang (from ports), to see if the problem goes away? > > I did try removing all custom make.conf settings (previously I just had > CPUTYPE?=icelake-server), but that didn't change the behavior. > > Before I try further build tweaks, I'd like to ask if the IOMMU fault > report can provide guidance here? AFAICT all the faults I'm getting > show "reason 0x3". If I'm reading the VT-d spec correctly, FR=0x3 > indicates an invalid context entry, in other words there's something the > hardware doesn't like in the way the address width or pagetable base is > configured for the PCIe requestor. I would start looking at the other direction: might be, there are still some left shifts for int32 values with the shift count > 30, or uint32 with the count > 31. Also might be useful to dump each context entry on creation, it is kept constant after.