From nobody Wed Sep 08 12:50:37 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0675817B6114 for ; Wed, 8 Sep 2021 12:50:42 +0000 (UTC) (envelope-from se@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4H4MSK5TYtz3mJF; Wed, 8 Sep 2021 12:50:41 +0000 (UTC) (envelope-from se@freebsd.org) Received: from [IPV6:2003:cd:5f18:5600:5449:cc68:8dc4:e518] (p200300cd5f1856005449cc688dc4e518.dip0.t-ipconnect.de [IPv6:2003:cd:5f18:5600:5449:cc68:8dc4:e518]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 26CB49E41; Wed, 8 Sep 2021 12:50:41 +0000 (UTC) (envelope-from se@freebsd.org) Message-ID: Date: Wed, 8 Sep 2021 14:50:37 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.1.0 Subject: Re: -CURRENT compilation time Content-Language: en-US To: David Chisnall References: <2cfb912a-618b-9f06-9cef-d2fe1d78fe97@FreeBSD.org> From: Stefan Esser Cc: freebsd-current@freebsd.org In-Reply-To: <2cfb912a-618b-9f06-9cef-d2fe1d78fe97@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------52cROGg8SECfscciZvLSd0YI" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------52cROGg8SECfscciZvLSd0YI Content-Type: multipart/mixed; boundary="------------i8YLwYeOMxFAkQNdHxy8K044"; protected-headers="v1" From: Stefan Esser To: David Chisnall Cc: freebsd-current@freebsd.org Message-ID: Subject: Re: -CURRENT compilation time References: <2cfb912a-618b-9f06-9cef-d2fe1d78fe97@FreeBSD.org> In-Reply-To: <2cfb912a-618b-9f06-9cef-d2fe1d78fe97@FreeBSD.org> --------------i8YLwYeOMxFAkQNdHxy8K044 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Am 08.09.21 um 10:57 schrieb David Chisnall: > On 07/09/2021 18:02, Stefan Esser wrote: >> Wouldn't this break META_MODE? >=20 > I have never managed to get META_MODE to work but my understanding is t= hat > META_MODE is addressing a problem that doesn't really exist in any othe= r build > system that I've used: that dependencies are not properly tracked. META_MODE allows for complex interdependencies. They are no issue in the GPL/Linux world, since components are not integrated in the same way as has been practice in BSD for many decades. > When I do a build of LLVM with the upstream build system with no change= s, it > takes Ninja approximately a tenth of a second to stat all of the releva= nt files > and tell me that I have no work to do.=C2=A0 META_MODE apparently lets = the FreeBSD > build system extract these dependencies and do something similar, but i= t's not > enabled by default and it's difficult to make work. I tend to disagree on the last 5 words of your last sentence. It took me just a few seconds to activate, and it has worked without faul= t since. There are only 2 trivial steps. But it is easy to miss the fact, that WITH_META_MODE has to be added to /etc/src-env.conf, not /etc/src.conf: 1) Add "WITH_META_MODE=3Dyes" to /etc/src-env.conf (create file, if it do= es not exist) 2) Add "device filemon" to your kernel configuration or to the kld_load variable in /etc/rc.conf to load the kernel module (The kernel module can of course also be manually loaded at any time.) >> I'd rather be able to continue building the world within a few minutes= >> (generally much less than 10 minutes, as long as there is no major LLV= M >> upgrade) than have a faster LLVM build and then a slower build of the = world ... >=20 > The rest of this thread has determined that building LLVM accounts for = half of > the build time in a clean FreeBSD build.=C2=A0 LLVM's CMake is not a gr= eat example: > it has been incrementally improved since CMake 2.8 and doesn't yet use = any of > the modern CMake features that allow encapsulating targets and providin= g import > / export configurations. The build of LLVM is skipped if META_MODE is enabled, except if there really was a change to some LLVM header that causes a complete rebuild. A further speed-up can be had with ccache, but I found that it does not seem to make that much of a difference on my system. > In spite of that, it generates a ninja file that compiles *significantl= y* > faster than the bmake-based system in FreeBSD.=C2=A0 In other projects = that I've > worked on with a similar-sized codebase to FreeBSD that use CMake + Nin= ja, I've > never had the same problems with build speed that I have with FreeBSD. Possible, but if I watch the LLVM build with top or systat, I see that all my cores are busy, nearly throughout the full build. There are two methods that could theoretically speed-up the build: 1) make use of idle CPU cores 2) reduce the number of object files to build I do not see that there is much potential for 1), since there is a high degree of parallelism: >>> World build completed on Wed Sep 1 13:40:14 CEST 2021 >>> World built in 99 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- 98.69 real 741.61 user 234.55 sys >>> World build completed on Thu Sep 2 23:22:04 CEST 2021 >>> World built in 98 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- 98.34 real 780.41 user 228.67 sys >>> World build completed on Fri Sep 3 19:09:39 CEST 2021 >>> World built in 165 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- 164.84 real 1793.62 user 241.11 sys >>> World build completed on Sun Sep 5 20:23:29 CEST 2021 >>> World built in 135 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- 135.59 real 695.45 user 214.76 sys >>> World build completed on Mon Sep 6 21:10:44 CEST 2021 >>> World built in 478 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- 479.22 real 11374.40 user 474.19 sys >>> World build completed on Wed Sep 8 11:51:03 CEST 2021 >>> World built in 652 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- 652.14 real 17857.03 user 753.41 sys Calculating "(user + sys) / real" I get factors between 10 (in case of only minor changes) to 28 for larger recompiles (e.g. if lots of source files depend on an updated header), with 32 the theoretical limit for all cores continuously active during the build. META_MODE does not understand that updated build tools do not always require a full rebuild, but special cases have been added to the Makefile to reduce the number of unnecessary rebuilds. > Working on LLVM, I generally spend well under 10% of my time either wai= ting for > builds or fighting the build system.=C2=A0 Working on FreeBSD, I genera= lly spend > over 90% of my time waiting for builds or fighting the build system.=C2= =A0 This > means that my productivity contributing to FreeBSD is almost zero. >=20 > For reference, changes to LLVM typically build for me in under 30 secon= ds with > Ninja, unless I've changed a header that everything You should get the same effect from META_MODE. And META_MODE allows to execute make in any subtree of a larger source tree, provided there is a Makefile for that part of the sources, e.g.: $ cd /usr/src/usr.bin/clang/lld $ make -j 32 [...] $ touch /usr/src/contrib/llvm-project/lld/Common/Strings.cpp $ time make -j 32 Building /usr/obj/usr/git/src/amd64.amd64/usr.bin/clang/lld/Common/String= s.o Building /usr/obj/usr/git/src/amd64.amd64/usr.bin/clang/lld/ld.lld.full Building /usr/obj/usr/git/src/amd64.amd64/usr.bin/clang/lld/ld.lld.debug Building /usr/obj/usr/git/src/amd64.amd64/usr.bin/clang/lld/ld.lld real 0m1.699s user 0m1.454s sys 0m3.745s This assumes that your world is up-to-date in general, which should be the case when working on a single component, e.g. one of the programs belonging to CLANG/LLVM. Compiling a single file and linking a single target does of course not allow for any parallelism. $ cd /usr/src/usr.bin/clang $ time make -j 32 > /dev/null real 0m24.650s user 8m58.381s sys 0m34.786s Some files have changed between the last "make buildworld" and now, but it takes less than 10 minutes of CPU time (i.e. 1 minute real time on a system with 6 cores / 12 threads) to update the obj directory. My system got an overall parallelism of 24 on that run ... And thereafter, there is hardly any overhead caused by bmake: $ time make -j 32 > /dev/null real 0m0.064s user 0m0.453s sys 0m0.029s The overhead for scanning all LLVM components if there was no change is absolutely negligible. Now lets touch a header that is included in a number of files: $ touch /usr/src/contrib/llvm-project/lld/include/lld/Common/Strings.h $ time make -j 32 | wc 41 82 3130 real 0m17.727s user 2m56.021s sys 0m8.237s The ratio of user+sys to real is a little above 10, which is not too bad if you consider that the link phase needs to wait for all object files. I really do not see that there is much to gain, here ... > In particular, building FreeBSD on a 10-24 core machine has very long p= eriods > where a number of the cores are completely idle. I do not observe this, and I'm using a 16 core / 32 thread CPU, which would be busy on all cores with just 16 parallel threads, just not taking advantage of SMT. And I see an overall factor (and load average) of 10 for small changes (with lots of I/O and little processing), 28 and beyond if larger parts of the system need to be rebuild. (BTW: This is a system with SATA drives in a RAIDZ1 configuration, which limits the I/O rate way below what an SSD based system might get.) > Ninja also has a few other nice features that improve performance relat= ive to > bmake: >=20 > =C2=A0- It lets you put jobs in different pools.=C2=A0 In LLVM this is = used to put link > and compile jobs in different pools because linking with LLD uses multi= ple > threads and a lot more memory than compilation, so a 10-core machine ma= y want > to do 12 compile jobs in parallel but only 2 link jobs.=C2=A0 This make= s it much > easier to completely saturate the machine. That may be an advantage on systems with relatively small RAM (compared t= o the number of threads supported in parallel). But I do see compilations and linking executing in parallel, i.e. just on= e linker activation while compiles run in parallel. Due to the multi-thread= ed execution of LLD that might cause the load average to slightly exceed the= number of threads, but only for a relatively short period of time. > =C2=A0- Ninja provides each parallel build task with a separate pipe fo= r stdout and > stderr, and does not print their output unless a build step fails (or u= nless > you build with -v).=C2=A0 With bmake, if a parallel build fails I have = to rerun the > build without -j, because the output is interleaved with succeeding job= s and > it's difficult to see what actually failed.=C2=A0 With ninja, the outpu= t is from > each failed job, with no interleaving. This makes it easier to spot the error line, but I've got used to the way= the output is formatted by a parallel make run. I always redirect the output into a file (either with nohup or with tee, depending on whether I plan to watch the build). I hardly ever have to us= e a single threaded build to spot an error, it is generally obvious from th= e build log in that file. But I do agree that separate output per build job makes this easier, just= not enough that I'd want to rework the whole build system. And especially= not if it breaks META_MODE. Regards, STefan --------------i8YLwYeOMxFAkQNdHxy8K044-- --------------52cROGg8SECfscciZvLSd0YI Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEo3HqZZwL7MgrcVMTR+u171r99UQFAmE4sZ0FAwAAAAAACgkQR+u171r99URf yggAsMj51OT20hc/Xx+6LjiohXJh9AXXHO3JCfp/LFguuGSYl5zWfzL34oxsrEh10oQA1Dv+wGPX iAfpoZEjnd8KDH9KQOBgrziLmAYsDQ0bsIOgzlc8T07GmYqaGBgAzhf9JOeuowElhOf3wcy3CTi8 U+UwK5keUmcylBOnHYwdTZXlS5aaAW75y7iO9UrKbLFMV9j2Ge7MocOhOpUCf2P3GKBK2IUL5bHI Zr2NxlbmWolE6dvBn7a9bGOpoCL/8WHAHliZI8z7Dc+tr8Yw80ZFvLiTDpd0yXz1UBJ/hidrOB9L C3XcuEq5XHEKilfpgZzJfS4VBhxL5OtSvj2ogFYyYQ== =DHeo -----END PGP SIGNATURE----- --------------52cROGg8SECfscciZvLSd0YI--