Buildworld times (was Re: svn commit: r350256 - in stable/12: . contrib/compiler-rt/lib/sanitizer_common contrib/libunwind/src contrib/llvm/lib/DebugInfo/DWARF contrib/llvm/lib/MC contrib/llvm/lib/Object contrib/llvm/lib/Ta...)

Mon Jul 29 17:44:05 UTC 2019

On 7/26/2019 10:38 PM, Jeremy Chadwick via freebsd-stable wrote:
> (Please retain CCs, I am not subscribed to the list)
>
> Below is hard evidence of 3 things on stable/11 (not 12) after r350259:
>
> 1. r350259 adds *substantial* time to buildworld.

Are you sure this is not the same as the issue in RELENG12 ? ie. the new
version of clang is built as part of world since it differs from whats
installed.  I had a RELENG11 box sitting around from July 4th

(nanobsd2)# time make -j8 buildworld > /var/log/build.out
13421.587u 906.968s 1:01:36.13 387.6%   75193+900k 285907+3240194io
108879pf+0w
0(nanobsd2)#

Updated to revision 350414.
0(nanobsd2)# time make -j8 buildworld > /var/log/build.out
18656.513u 1117.246s 1:23:53.24 392.8%  70122+916k 255703+3772612io
104343pf+0w
0(nanobsd2)#

Did installworld/kernel.

buildworld after that

0(nanobsd2)# time make -j8 buildworld > /var/log/build.out
12998.964u 649.697s 58:48.61 386.7%     75748+907k 358314+3247016io
108534pf+0w
0(nanobsd2)#

Which is probably about the same as before the update build time wise

> 2. WITHOUT_CLANG_EXTRAS+WITHOUT_CLANG_FULL+WITHOUT_LLDB can help improve
> the situation after r350259, but it is still no where near as fast as
> pre-r350259.
>
> 3. Kernel build times are fine; issue is with world.
>
> TL;DR for lazy folks:
>
> stable/11 r350330 world + minimal clang = 1:29:34
> stable/11 r350330 world + full clang    = 1:46:31
> stable/11 r350252 world + minimal clang =   56:52
> stable/11 r350252 world + full clang    = 1:14:30
>
> I cannot even begin to tell you how big of an impact this has on my
> low-end dual-core VPS box (world takes hours upon hours).
>
> We've been down this road before, many many times, since the
> introduction of clang/LLVM.  Here's just a few that went no where.  I
> couldn't find the more-useful one that had some concrete numbers in it,
> dating back to pre-2016 (maybe sometime in 2014 or 2015?):
>
> https://lists.freebsd.org/pipermail/freebsd-current/2017-January/thread.html#64431
> https://lists.freebsd.org/pipermail/freebsd-stable/2017-January/thread.html#86646
> https://lists.freebsd.org/pipermail/freebsd-questions/2016-November/thread.html#274684
>
> Does anyone have a good/recent write-up on how to switch to gcc?  :-)
>
>
> System
> ======
> * Intel Core 2 Quad Q9550 @ 2.83GHz
> * 8GB ECC RAM
> * Samsung SSD 840 EVO 250GB filesystem (UFS2 + SU (not SUJ) + TRIM) + 32GB swap
> * Running stable/11 r349226
> * Misc notes
>   - r350330 happened to be what was "master" at the time of my test
>   - r350252 was the commit on stable/11 immediately before r350259
>   - Switching to r350252 accomplished via: cd /usr/src && svnlite up -r350252
>   - System uses kern.maxvnodes=856944, last tuned 2018/06/07
>
>
> Test #1, building r350330 minimal clang
> =======================================
> # cat /etc/src.conf
> WITHOUT_ATM=true
> WITHOUT_BLUETOOTH=true
> WITHOUT_DEBUG_FILES=true
> WITHOUT_FLOPPY=true
> WITHOUT_FREEBSD_UPDATE=true
> WITHOUT_IPFILTER=true
> WITHOUT_IPX=true
> WITHOUT_LIB32=true
> WITHOUT_NDIS=true
> WITHOUT_NETGRAPH=true
> WITHOUT_PPP=true
> WITHOUT_SENDMAIL=true
> WITHOUT_TESTS=true
> WITHOUT_WIRELESS=true
> WITH_OPENSSH_NONE_CIPHER=true
> WITHOUT_CLANG_EXTRAS=true
> WITHOUT_CLANG_FULL=true
> WITHOUT_LLDB=true
> WITHOUT_LLVM_TARGET_AARCH64=true
> WITHOUT_LLVM_TARGET_ARM=true
> WITHOUT_LLVM_TARGET_MIPS=true
> WITHOUT_LLVM_TARGET_POWERPC=true
> WITHOUT_LLVM_TARGET_SPARC=true
> WITHOUT_REPRODUCIBLE_BUILD=true
> # cat /etc/make.conf
> KERNCONF=X7SBA_RELENG_11_amd64
> CPUTYPE?=core2
> SVN_UPDATE=yes
> STRIP=
> CFLAGS+=	-fno-omit-frame-pointer
>
> Result:
> # rm -fr /usr/obj/*
> # cd /usr/src
> # time make -j4 buildworld
> 19906.874u 1280.928s 1:29:33.51 394.3%  57966+778k 23504+14200io 13867pf+0w
> # time make -j4 buildkernel
> 1592.460u 196.047s 7:36.61 391.6%       48704+614k 6627+18158io 7361pf+0w
>
>
> Test #2, building r350330 full clang
> ====================================
> "full clang" means same as Test #1 but with these 3 src.conf lines
> commented out, i.e. CLANG_EXTRAS, CLANG_FULL, and LLDB are ENABLED:
>
> WITHOUT_CLANG_EXTRAS=true
> WITHOUT_CLANG_FULL=true
> WITHOUT_LLDB=true
>
> Result:
> # rm -fr /usr/obj/*
> # cd /usr/src
> # time make -j4 buildworld
> 23779.674u 1463.156s 1:46:30.75 394.9%  57621+783k 20093+15423io 7283pf+0w
> # time make -j4 buildkernel
> 1594.079u 194.345s 7:36.48 391.7%       48707+614k 5301+18013io 5342pf+0w
>
>
> Test #3, building r350252 minimal clang
> =======================================
> Same configs as Test #1
>
> Result:
> # rm -fr /usr/obj/*
> # cd /usr/src
> # time make -j4 buildworld
> 12582.693u 882.543s 56:52.35 394.6%     62698+760k 21432+9694io 6923pf+0w
> # time make -j4 buildkernel
> 1649.559u 184.934s 7:48.01 391.9%       57053+622k 7566+18291io 5402pf+0w
>
>
> Test #4, building r350252 full clang
> ====================================
> Same configs as Test #2
>
> # rm -fr /usr/obj/*
> # cd /usr/src
> # time make -j4 buildworld
> 16600.975u 1068.754s 1:14:29.53 395.3%  63271+774k 8683+10876io 4707pf+0w
> # time make -j4 buildkernel
> 1650.654u 183.966s 7:47.47 392.4%       57117+623k 2829+17951io 1926pf+0w
>