From nobody Wed Sep 08 08:57:50 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2097117B8F70 for ; Wed, 8 Sep 2021 08:57:53 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4H4GHj0Stgz3G0G; Wed, 8 Sep 2021 08:57:53 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id DED7082A9; Wed, 8 Sep 2021 08:57:52 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from [192.168.1.202] (host86-148-130-214.range86-148.btcentralplus.com [86.148.130.214]) by smtp.theravensnest.org (Postfix) with ESMTPSA id E07EC2C94B; Wed, 8 Sep 2021 09:57:51 +0100 (BST) Subject: Re: -CURRENT compilation time To: Stefan Esser Cc: freebsd-current@freebsd.org References: From: David Chisnall Message-ID: <2cfb912a-618b-9f06-9cef-d2fe1d78fe97@FreeBSD.org> Date: Wed, 8 Sep 2021 09:57:50 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N On 07/09/2021 18:02, Stefan Esser wrote: > Wouldn't this break META_MODE? I have never managed to get META_MODE to work but my understanding is that META_MODE is addressing a problem that doesn't really exist in any other build system that I've used: that dependencies are not properly tracked. When I do a build of LLVM with the upstream build system with no changes, it takes Ninja approximately a tenth of a second to stat all of the relevant files and tell me that I have no work to do. META_MODE apparently lets the FreeBSD build system extract these dependencies and do something similar, but it's not enabled by default and it's difficult to make work. > I'd rather be able to continue building the world within a few minutes > (generally much less than 10 minutes, as long as there is no major LLVM > upgrade) than have a faster LLVM build and then a slower build of the world ... The rest of this thread has determined that building LLVM accounts for half of the build time in a clean FreeBSD build. LLVM's CMake is not a great example: it has been incrementally improved since CMake 2.8 and doesn't yet use any of the modern CMake features that allow encapsulating targets and providing import / export configurations. In spite of that, it generates a ninja file that compiles *significantly* faster than the bmake-based system in FreeBSD. In other projects that I've worked on with a similar-sized codebase to FreeBSD that use CMake + Ninja, I've never had the same problems with build speed that I have with FreeBSD. Working on LLVM, I generally spend well under 10% of my time either waiting for builds or fighting the build system. Working on FreeBSD, I generally spend over 90% of my time waiting for builds or fighting the build system. This means that my productivity contributing to FreeBSD is almost zero. For reference, changes to LLVM typically build for me in under 30 seconds with Ninja, unless I've changed a header that everything In particular, building FreeBSD on a 10-24 core machine has very long periods where a number of the cores are completely idle. Ninja also has a few other nice features that improve performance relative to bmake: - It lets you put jobs in different pools. In LLVM this is used to put link and compile jobs in different pools because linking with LLD uses multiple threads and a lot more memory than compilation, so a 10-core machine may want to do 12 compile jobs in parallel but only 2 link jobs. This makes it much easier to completely saturate the machine. - Ninja provides each parallel build task with a separate pipe for stdout and stderr, and does not print their output unless a build step fails (or unless you build with -v). With bmake, if a parallel build fails I have to rerun the build without -j, because the output is interleaved with succeeding jobs and it's difficult to see what actually failed. With ninja, the output is from each failed job, with no interleaving. David