From nobody Tue Feb 21 12:55:33 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PLfXM5MWvz3sbb6 for ; Tue, 21 Feb 2023 13:00:19 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PLfXM1pJtz4GqV for ; Tue, 21 Feb 2023 13:00:19 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 31LD05hV036944 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 21 Feb 2023 14:00:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 31LD05AS036942; Tue, 21 Feb 2023 14:00:05 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.16.1/8.16.1) with ESMTPS id 31LCvhQl077018 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Tue, 21 Feb 2023 13:57:44 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 31LCtX5M078942 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 21 Feb 2023 13:55:35 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 31LCtXTl078941; Tue, 21 Feb 2023 13:55:33 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Tue, 21 Feb 2023 13:55:33 +0100 From: Peter To: Mark Millard Cc: FreeBSD-STABLE Mailing List Subject: Re: 13.2 BETA2: how do debug META_MODE? Message-ID: References: <41B536B0-DA66-449E-96BB-E11A8750471A.ref@yahoo.com> <41B536B0-DA66-449E-96BB-E11A8750471A@yahoo.com> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <41B536B0-DA66-449E-96BB-E11A8750471A@yahoo.com> X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Tue, 21 Feb 2023 14:00:08 +0100 (CET) X-Rspamd-Queue-Id: 4PLfXM1pJtz4GqV X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mon, Feb 20, 2023 at 08:44:59PM -0800, Mark Millard wrote: ! Peter wrote on ! Date: Tue, 21 Feb 2023 03:45:12 UTC : ! ! > on /some/ of my nodes, META_MODE seems not being honored anymore: ! > I had to build them another time, and the lengthy lib/clang gets ! > built all over again (tried two times). ! > This is so since 13.2 (BETA2). It did work in 13.1 (RELENG), at least ! > according to the timing from the logfiles. ! > ! > Now I'm trying to figure out the difference, because I have some ! > nodes where it appears to more-or-less work (have seen buildworld ! > take 5 minutes), and others where it doesn't (take an hour to build). ! > The thing is scripted, so it is not so very likely an operator error ! > (while not impossible either). ! > ! > But it seems difficult to figure out details: "make -n" seems to not ! > care about META_MODE, while META_MODE suppresses all useful output from ! > make. And the docs say there are *.meta files (yes there are), but no ! > info about how to verify their content, or how to get make tell what ! > it is going to do and why (and the buildworld is not the most easy ! > to understand target)... ! > ! > So, some inspiration would be welcome... ! ! On thing to check on is if filemon.ko is loaded and operational. ! META_MODE greatly depends on it. That should be the case - 'kldstat' shows it (and I've seen warnings where it didn't). ! Another thing to know is that the following are very different ! for what all is built for the "(again #0)" line vs. the other ! two "again" lines, using buildworld as an example context. ! Imagine here the the first buildworld rebuilds llvm/clang ! materials. ! ! # cd /usr/src/ ! # env WITH_META_MODE=yes make buildworld ! # env WITH_META_MODE=yes make installworld ! # env WITH_META_MODE=yes make buildworld (again #0) ! ## no more rebuilds below? ! # env WITH_META_MODE=yes make buildworld (again #1) ! # env WITH_META_MODE=yes make buildworld (again #2) But what is the difference between #0 and #1? I am actually grabbing src and obj and zfs-send them away to the actual install target, then installworld to DESTDIR which becomes the new root for the next update cycle. Jails are a bit different because not all of them get the compiler installed - and jails do have the problem now: if they get the compiler, it always rebuilt. According to the ./UPDATING paper, one should do build + install + build-again + install-again. That is probably outdated, but on Release-Changes I'm doing it that way. During the build-again step I noticed that the META_MODE did already work, things were fast - but only for the base nodes, not for the jails. ! Unfortunately, the some of the install activity registers ! as activity that is to cause later rebuild activity: ! updated file dates. Okay. I tried to find the offensive timestamps, and the problem is: there aren't any. :( I grabbed one node that does build fast as expected, and one that doesn't, and put the single-core output of buildworld side-by-side. and this is where the difference starts: [fast builds] ===> kerberos5/tools/make-roken (obj,all,install) Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken.c Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken.o Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/_proginstall ===> kerberos5/lib/libroken (obj,all,install) Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/libroken.a building static roken library Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/_libinstall Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/_INCSINS ===> kerberos5/lib/libvers (obj,all,install) Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libvers/libvers.a building static vers library [rebuilds it all] ===> kerberos5/tools/make-roken (obj,all,install) Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken.c Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken.o Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/_proginstall ===> kerberos5/lib/libroken (obj,all,install) Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/roken.h Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/base64.o Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/copyhostent.o Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/ecalloc.o Building /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/emalloc.o ... Looking at the first concerned file, /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/roken.h and it's *.meta file, the timestamps aren't really different. In fact, those from the *.meta file are all in the future - the question seems to become rather: why did that ever work? ;) # Meta data file /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/roken.h.meta CMD make-roken > roken.h CWD /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken TARGET roken.h -- command output -- -- filemon acquired metadata -- # filemon version 5 # Target pid 1754 # Start 1676683461.944305 V 5 E 1757 /bin/sh R 1757 /etc/libmap.conf R 1757 /var/run/ld-elf.so.hints R 1757 /lib/libedit.so.8 R 1757 /lib/libc.so.7 R 1757 /lib/libncursesw.so.9 R 1757 /usr/share/locale/en_US.UTF-8/LC_CTYPE F 1757 1758 W 1758 roken.h E 1758 /usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin/make-roken X 1758 0 0 X 1757 0 0 # Stop 1676683461.954451 # Bye bye The /usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin/make-roken in this output is a different one than what was built right beforehand (in fact there are three or four of them). ! There are also issues of sort of a feedback loop: rm ! ends up updated (deleted and replaced) by install but rm ! was also listed as part of the sequence of replacing some ! other files. Result? The rm removal/replacement ends up ! meaning the files are to be regenerated, not just recopied. ! There is a long list of such commands, not just rm. ! ! "again #0" will rebuild llvm/clang. The other two "again"s ! will not. Strangely, it didn't, up to now. And it still doesn't on the base nodes. And as I am just juggling around with zfs clones, and fixup the src timestamps from the commitlogs anyway, I could probably fixup this also - if I only get to what is actually happening. ! See: ! ! https://lists.freebsd.org/pipermail/freebsd-current/2021-January/078488.html ! ! and: ! ! https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257616 . Thank You, that's exactly the inspiration I was looking for! Diving back in...