make buildworld is now 50% slower

[Daniel Braniss]

> On Sun, Jul 07, 2013 at 11:50:29AM +0300, Daniel Braniss wrote:
> > > On Fri, Jul 05, 2013 at 02:39:00PM +0200, Dimitry Andric wrote:
> > > > [redirecting to the correct mailing list, freebsd-stable@ ...]
> > > > 
> > > > On Jul 5, 2013, at 10:53, Daniel Braniss <danny at cs.huji.ac.il> wrote:
> > > > > after today's update of 9.1-STABLE I noticed that make build[world|kernel] are
> > > > > taking conciderable more time, is it because the upgrade of clang?
> > > > > and if so, is the code produced any better?
> > > > > 
> > > > > before:
> > > > > buildwordl:	 26m4.52s real 2h28m32.12s user 36m6.27s sys
> > > > > buildkernel:	 7m29.42s real 23m22.22s user 4m26.26s sys
> > > > > 
> > > > > today:
> > > > > buildwordl:	34m29.80s real 2h38m9.37s user 37m7.61s sys
> > > > > buildkernel:    15m31.52s real 22m59.40s user 4m33.06s sys
> > > > 
> > > > Ehm, your user and sys times are not that much different at all, they
> > > > add up to about 5% slower for buildworld, and 1% faster for build kernel.
> > > > Are you sure nothing else is running on that machine, eating up CPU time
> > > > while you are building? :)
> > > > 
> > > > But yes, clang 3.3 is of course somewhat larger than 3.2.  You might
> > > > especially notice that, if you are using gcc, which is very slow at
> > > > compiling C++.
> > > > 
> > > > In any case, if you do not care about clang, just set WITHOUT_CLANG= in
> > > > your /etc/src.conf, and you can shave off some build time.
> > > 
> > > I just built world/kernel (stable/9 r252769) 5 hours ago.  Results:
> > > 
> > > time make -j4 buildworld  = roughly 21 minutes on my hardware
> > > time make -j4 buildkernel = roughly 8 minutes on my hardware
> > > 
> > 
> > It's been a long time since I saw such numbers, maybe it's time
> > to see where time is being spent, I will run it without clang to compare with
> > your numbers.
> > 
> > > These numbers are about the norm for me, meaning I do not see a
> > > substantial increase in build times.
> > > 
> > > Key point: I do not use/build/grok clang, i.e. WITHOUT_CLANG=true is in
> > > my src.conf.  But I am aware of the big clang change in r252723.
> > > 
> > > If hardware details are wanted, ask, but I don't think it's relevant to
> > > what the root cause is.
> > > 
> > 
> > from what you are saying, I guess clang is not responsible.
> > looking for my Sherlock Holmes hat.
> 
> Some points to those numbers I stated above:
> 
> - System is an Intel Q9550 with 8GB of RAM
> 
> - Single SSD (UFS2+SU+TRIM) is used for root, /usr, /var, /tmp, and swap
> 
> - /usr/src is on ZFS (raidz1 + 3 disks) -- however I got equally small
> numbers when it was on the SSD
> 
> - /usr/src is using compression=lz4  (to folks from -fs: yeah, I'm
> trying it out to see how much of an impact it has on interactivity.  I
> can still tell when it kicks in, but it's way, way better than lzjb.
> Rather not get into that here)
> 
> - Contents of /etc/src.conf (to give you some idea of what I disable):
> 
> WITHOUT_ATM=true
> WITHOUT_BLUETOOTH=true
> WITHOUT_CLANG=true
> WITHOUT_FLOPPY=true
> WITHOUT_FREEBSD_UPDATE=true
> WITHOUT_INET6=true
> WITHOUT_IPFILTER=true
> WITHOUT_IPX=true
> WITHOUT_KERBEROS=true
> WITHOUT_LIB32=true
> WITHOUT_LPR=true
> WITHOUT_NDIS=true
> WITHOUT_NETGRAPH=true
> WITHOUT_PAM_SUPPORT=true
> WITHOUT_PPP=true
> WITHOUT_SENDMAIL=true
> WITHOUT_WIRELESS=true
> WITH_OPENSSH_NONE_CIPHER=true
> 
> It's WITHOUT_CLANG that cuts down the buildworld time by a *huge* amount
> (I remember when it got introduced, my buildworld jumped up to something
> like 40 minutes); the rest probably save a minute or two at most.
> 
> - /etc/make.conf doesn't contain much that's relevant, other than:
> 
> CPUTYPE?=core2
> 
> # For DTrace; also affects ports
> STRIP=
> CFLAGS+=	-fno-omit-frame-pointer
> 
> - I do some tweaks in /etc/sysctl.conf (mainly vfs.read_min and
> vfs.read_max), but I will admit I am not completely sure what those
> do quite yet (I just saw the commit from scottl@ a while back talking
> about how an increased vfs.read_min helps them at Netflix quite a
> lot).  I also adjust kern.maxvnodes.
> 
> - Some ZFS ARC settings are adjusted in /boot/loader.conf (I'm playing
> with some stuff I read in Andriy Gapon's ZFS PDF), but they definitely
> do not have a major impact on the numbers I listed off.
> 
> - I do increase kern.maxdsiz, kern.dfldsiz, and kern.maxssiz in
> /boot/loader.conf to 2560M/2560M/256M respectively, but that was mainly
> from the days when I ran MySQL and needed a huge userland processes.
> 
> All in all my numbers are low/small because of two things: the SSD, and
> WITHOUT_CLANG.
> 
> Hope this gives you somewhere to start/stuff to ponder.
>
indeed!

on my pretty much standard dev machine, PowerEdge R710, X5550  @ 2.67GHz, 
16384 MB,
root on ufs, the rest is via ZFS, this is what I'm getting for buildworld:
with clang	27m32.91s real 2h42m52.82s user 36m20.69s sys
without clang	13m24.19s real 1h23m26.52s user 29m5.18s sys

on a similar machine but with root via NFS, and /var on ZFS, the numbers are 
similar:
with clang	23m30.92s real 2h9m8.85s user 29m27.84s sys
without clang	12m7.53s real 1h7m54.24s user 23m9.78s sys
(this host is newer, PowerEdge C6220, E5-2643 0 @ 3.30GHz, 32768 MB)

it's still not clear what happened last week, but probably my machine had some
runaway process.

cheers,
	danny