Buildworld times (was Re: svn commit: r350256 - in stable/12: . contrib/compiler-rt/lib/sanitizer_common contrib/libunwind/src contrib/llvm/lib/DebugInfo/DWARF contrib/llvm/lib/MC contrib/llvm/lib/Object contrib/llvm/lib/Ta...)

Sat Jul 27 02:43:46 UTC 2019

(Please retain CCs, I am not subscribed to the list)

Below is hard evidence of 3 things on stable/11 (not 12) after r350259:

1. r350259 adds *substantial* time to buildworld.

2. WITHOUT_CLANG_EXTRAS+WITHOUT_CLANG_FULL+WITHOUT_LLDB can help improve
the situation after r350259, but it is still no where near as fast as
pre-r350259.

3. Kernel build times are fine; issue is with world.

TL;DR for lazy folks:

stable/11 r350330 world + minimal clang = 1:29:34
stable/11 r350330 world + full clang    = 1:46:31
stable/11 r350252 world + minimal clang =   56:52
stable/11 r350252 world + full clang    = 1:14:30

I cannot even begin to tell you how big of an impact this has on my
low-end dual-core VPS box (world takes hours upon hours).

We've been down this road before, many many times, since the
introduction of clang/LLVM.  Here's just a few that went no where.  I
couldn't find the more-useful one that had some concrete numbers in it,
dating back to pre-2016 (maybe sometime in 2014 or 2015?):

https://lists.freebsd.org/pipermail/freebsd-current/2017-January/thread.html#64431
https://lists.freebsd.org/pipermail/freebsd-stable/2017-January/thread.html#86646
https://lists.freebsd.org/pipermail/freebsd-questions/2016-November/thread.html#274684

Does anyone have a good/recent write-up on how to switch to gcc?  :-)

System
======
* Intel Core 2 Quad Q9550 @ 2.83GHz
* 8GB ECC RAM
* Samsung SSD 840 EVO 250GB filesystem (UFS2 + SU (not SUJ) + TRIM) + 32GB swap
* Running stable/11 r349226
* Misc notes
  - r350330 happened to be what was "master" at the time of my test
  - r350252 was the commit on stable/11 immediately before r350259
  - Switching to r350252 accomplished via: cd /usr/src && svnlite up -r350252
  - System uses kern.maxvnodes=856944, last tuned 2018/06/07

Test #1, building r350330 minimal clang
=======================================
# cat /etc/src.conf
WITHOUT_ATM=true
WITHOUT_BLUETOOTH=true
WITHOUT_DEBUG_FILES=true
WITHOUT_FLOPPY=true
WITHOUT_FREEBSD_UPDATE=true
WITHOUT_IPFILTER=true
WITHOUT_IPX=true
WITHOUT_LIB32=true
WITHOUT_NDIS=true
WITHOUT_NETGRAPH=true
WITHOUT_PPP=true
WITHOUT_SENDMAIL=true
WITHOUT_TESTS=true
WITHOUT_WIRELESS=true
WITH_OPENSSH_NONE_CIPHER=true
WITHOUT_CLANG_EXTRAS=true
WITHOUT_CLANG_FULL=true
WITHOUT_LLDB=true
WITHOUT_LLVM_TARGET_AARCH64=true
WITHOUT_LLVM_TARGET_ARM=true
WITHOUT_LLVM_TARGET_MIPS=true
WITHOUT_LLVM_TARGET_POWERPC=true
WITHOUT_LLVM_TARGET_SPARC=true
WITHOUT_REPRODUCIBLE_BUILD=true
# cat /etc/make.conf
KERNCONF=X7SBA_RELENG_11_amd64
CPUTYPE?=core2
SVN_UPDATE=yes
STRIP=
CFLAGS+=	-fno-omit-frame-pointer

Result:
# rm -fr /usr/obj/*
# cd /usr/src
# time make -j4 buildworld
19906.874u 1280.928s 1:29:33.51 394.3%  57966+778k 23504+14200io 13867pf+0w
# time make -j4 buildkernel
1592.460u 196.047s 7:36.61 391.6%       48704+614k 6627+18158io 7361pf+0w

Test #2, building r350330 full clang
====================================
"full clang" means same as Test #1 but with these 3 src.conf lines
commented out, i.e. CLANG_EXTRAS, CLANG_FULL, and LLDB are ENABLED:

WITHOUT_CLANG_EXTRAS=true
WITHOUT_CLANG_FULL=true
WITHOUT_LLDB=true

Result:
# rm -fr /usr/obj/*
# cd /usr/src
# time make -j4 buildworld
23779.674u 1463.156s 1:46:30.75 394.9%  57621+783k 20093+15423io 7283pf+0w
# time make -j4 buildkernel
1594.079u 194.345s 7:36.48 391.7%       48707+614k 5301+18013io 5342pf+0w

Test #3, building r350252 minimal clang
=======================================
Same configs as Test #1

Result:
# rm -fr /usr/obj/*
# cd /usr/src
# time make -j4 buildworld
12582.693u 882.543s 56:52.35 394.6%     62698+760k 21432+9694io 6923pf+0w
# time make -j4 buildkernel
1649.559u 184.934s 7:48.01 391.9%       57053+622k 7566+18291io 5402pf+0w

Test #4, building r350252 full clang
====================================
Same configs as Test #2

# rm -fr /usr/obj/*
# cd /usr/src
# time make -j4 buildworld
16600.975u 1068.754s 1:14:29.53 395.3%  63271+774k 8683+10876io 4707pf+0w
# time make -j4 buildkernel
1650.654u 183.966s 7:47.47 392.4%       57117+623k 2829+17951io 1926pf+0w

-- 
| Jeremy Chadwick                                 jdc at koitsu.org |
| UNIX Systems Administrator                      PGP 0x2A389531 |
| Making life hard for others since 1977.                        |