ULE vs 4BSD in RELENG_7

[Bruce Evans]

On Sun, 4 Nov 2007, Jeff Roberson wrote:

> On Sun, 4 Nov 2007, Gelsema, P (Patrick) - FreeBSD wrote:

>> w/o patch
>> hulk# time make -j8 buildkernel
>> 837.808u 138.167s 10:28.96 155.1%       6349+1349k 2873+7780io 303pf+0w
>> 
>> w patch
>> hulk# time make -j8 buildkernel
>> 838.554u 168.316s 10:52.10 154.4%       6263+1332k 6489+7791io 11pf+0w
>> 
>> I only understand the 10:52 seconds thing, but it looks like it's
>> stressing less and still getting things done.
>
> Well this doesn't look very good at all.  System time increased by 30 
> seconds!  Must be too many extra context switches.  The minimum slice value 
> is probably too small and you've got an average of 4 threads per-core.

838 seconds for a kernel build!  I'm unhappy with ~32 seconds for a
kernel build on a system with 1/2 or 1/4 as much CPU here, with ULE
still taking an extra 0.5 seconds.  Mine is not an equivalent kernel
build since it is for ~4.10 with ~5.2 userland (so that the benchmark
isn't affected by anything except the kernel).

4BSD:
"make depend":      8.12 real         5.09 user         1.34 sys
"make":            31.94 real        56.18 user         4.49 sys
ULE:
"make depend":      8.09 real         5.16 user         1.19 sys
"make":            32.12 real        55.72 user         4.91 sys

The kernel is about 2/3 as large as GENERIC, with no acpi and no modules.
The machine is a Turion X2 2GHz in i386 mode.  The buffer cache was 100%
warm.  All makes normal ones (no buildkernel or modules pessimizations)
with -j4.

Before recent vm changes, the real times were about 0.20 seconds smaller
for "make".

Building on a local file system gives real times about 2 seconds smaller
for "make".

Building a ~5.2 kernel on a local file system on this machine takes about
10 seconds longer (~42 seconds for 4BSD and ~43 seconds for ULE).

Building a ~5.2 world over nfs (with a cold cache) on this machine takes
about 825 seconds with 4BSD and 833 seconds with ULE.

-current building itself is much slower due to source bloat and gcc bloat:
For build a -current kernel with the same features on an Opteron X2 1.8GHz
in amd64 mode over an untuned nfs with an untuned (INVARIANTS) kernel:

4BSD (several months ago), gcc-3.4.6:
"make depend":    110.32 real        15.83 user        10.20 sys
"make":            86.43 real       128.17 user        19.05 sys
ULE, gcc-4.2.1:
"make depend":     83.34 real        15.39 user         8.67 sys
"make":           108.03 real       170.77 user        23.83 sys

The "make depend" times are not directly comparable.  The untuned nfs is
so slow that "make -j4 depend" takes 50-100 seconds no matter how many
kernels are built in parallel, due to dead time waiting for nfs, so I
build 6 kernels in parallel ("make depend" itself is mostly serial) on
the slow machine but not normally.  The "make" times are directly
comparable (they are for "make -j4" of a single kernel on 2-way SMP
systems).

Bruce