40% slowdown with dynamic /bin/sh

[Garance A Drosihn]

At 12:23 AM -0500 11/26/03, Michael Edenfield wrote:
>
>Just to provide some real-world numbers, here's what I got
>out of a buildworld:

I have reformatted the numbers that Michael reported,
into the following table:

>Static /bin/sh:         Dynamic /bin/sh:
>   real    385m29.977s     real    455m44.852s   => 18.22%
>   user    111m58.508s     user    113m17.807s   =>  1.18%
>   sys      93m14.450s     sys     103m16.509s   => 10.76%
>                           user+sys              =>  5.53%

Here are some buildworld numbers of my own, from my system.
In my case, I am running on a single Athlon MP2000, with a
gig of memory.  It does a buildworld without paging to disk.

Static sh, No -j:      Dynamic sh, No -j:
   real    84m31.366s     real    86m22.429s   =>  2.04%
   user    50m33.013s     user    51m13.080s   =>  1.32%
   sys     29m59.047s     sys     33m04.082s   => 10.29%
                          user+sys             =>  4.66%

Static sh, -j2:        Dynamic sh, -j2:
   real    92m38.656s     real    95m21.027s   =>  2.92%
   user    51m48.970s     user    52m29.152s   =>  1.29%
   sys     32m07.293s     sys     34m40.595s   =>  7.95%
                          user+sys             =>  3.84%

Buildworld, static, with no '-j',
                  executed /bin/sh  32,308 times.

Buildworld, static, with '-j2',
                  executed /bin/sh  32,802 times.

(I was expecting a much larger difference between the
number of /bin/sh's in a plain 'make buildworld' versus
'make -j2 buildworld'  But apparently I was wrong...).

On all attempts, I started out by doing:
     rm -Rf /usr/obj/usr/src/*
     sync ; sleep 1 ; sync ; sleep 1 ; sync

before doing the 'make' command.  I usually start up a 'script'
command to capture the output of buildworld, but I did not do
that for these tests.  All of the above buildworlds were typed
in at the console.  Plain console, no X11 running.  All are
running on a snapshot of -current as of sometime Tuesday.  All
are compiling the exact same /usr/src (left over from installing
that snapshot of -current).

Aside: building 5.1-"security" on this same hardware took
the following times:
   real    54m10.092s   [  71.03% ]
   user    41m39.121s   [  24.40% ]
   sys     10m20.325s   [ 210.69% ]

And those times *are* with 'script' running, as well as a
perl-script which I use to summarize "interesting" data from
the output of a buildworld.  So, those times include extra
overhead which is not included in the above buildworlds.
That's from a 'make -j3', and obviously has a static /bin/sh.
So, if you take that as the base, then the buildworld for
5.2-release (using *static* /bin/sh and -j2) will see the
performance hits that I put in brackets.  That probably seems
like a pretty horrifying hit, but remember that 5.1-release
did *not* build /rescue at all (not for me at least :-), and
that is probably a significant part of the increase.

I add all this just because it is easy to imagine that some
people will do "world-stones" of 5.1-release vs 5.2-release,
and might assume that the entire difference is due to /bin/sh
or the dynamic /bin & /sbin in general.  The above is not a
good enough benchmark to say *where* the time has gone, but
please consider the issue of building /rescue!

I have attempted to do a "decent" job at coming up with all
these numbers, but I'm sure they are not perfect.  It would
be good to use something other than "world-stones" as a
benchmark, but of course I'm not volunteering to do that...

For those who think I'm spoiled by fast hardware, please note
that all of the above has been done while doing just two
buildworlds and one buildkernel+installkernel on my sparc64
box (and that second buildworld is not done yet...).  So I
certainly am interested in how freebsd runs on "slower HW"!

-- 
Garance Alistair Drosehn            =   gad at gilead.netel.rpi.edu
Senior Systems Programmer           or  gad at freebsd.org
Rensselaer Polytechnic Institute    or  drosih at rpi.edu