svn commit: r187132 - head/usr.bin/make

Poul-Henning Kamp phk at phk.freebsd.dk
Mon Feb 2 23:40:13 PST 2009


In message <20090202220628.GA76833 at dragon.NUXI.org>, "David O'Brien" writes:


David,

I am disappointed.

You of all people here should know better than making such a mess
out of benchmarks.

First of all, you don't bother to even calculate a standard deviation
even though we have a neat tool that does that in the base system,
it's called "ministat", try it.

Second you totally bungle your data collection, by not eliminating
cache-effects.  It should be evident to anybody that when your
numbers always follow the pattern "high, low, low", that another
run is necessary (you need 3 for stddev) and the first one should
be discarded.

Poul-Henning

>The abbreviated results for local disk are:
>
>MAKE=make at r187921 /usr/bin/time -h $MAKE -j16 -Q buildworld >outfile 2>&1
>        25m38.07s real          1h30m6.59s user         45m37.27s sys
>        25m15.23s real          1h30m1.67s user         45m28.50s sys
>        25m22.65s real          1h30m8.85s user         45m32.10s sys
>        (1538.07 + 1515.23 + 1522.65)/3 = 1525.32 sec ave
>        [10% improvement would reduce build by 2m33s]

Standard deviation: 11.65 seconds

>MAKE=make at r187921 /usr/bin/time -h $MAKE -j16 -Q -s buildworld >outfile 2>&1
>        24m36.12s real     1h27m27.18s user     44m26.88s sys
>        24m34.77s real     1h27m27.82s user     44m24.56s sys
>        24m29.01s real     1h27m22.16s user     44m28.23s sys
>        (1476.12 + 1474.77 + 1469.01)/3 = 1473.30 sec ave
>        => 1473.30 / 1525.32 * 100 - 100 = -3.41% change in build time

standard deviation: 3.77 second

Difference at 95.0% confidence
        -52.0167 +/- 19.6298
        -3.41022% +/- 1.28694%
        (Student's t, pooled s = 8.6605)

>        [compared with below]
>        => 1473.30 / 1511.70 * 100 - 100 = -2.54% change in build time

Difference at 95.0% confidence
        38.4033 +/- 31.7193
        2.60662% +/- 2.15294%
        (Student's t, pooled s = 13.9942)

>MAKE=make at r187921 /usr/bin/time -h $MAKE -j16 buildworld >outfile 2>&1
>        25m3.95s real      1h27m40.02s user     45m19.12s sys
>        24m57.35s real     1h27m33.31s user     45m6.81s sys
>        25m33.81s real     1h27m39.68s user     44m50.02s sys
>        (1503.95 + 1497.35 + 1533.81)/3 = 1511.70 sec ave
>        => 1511.70 / 1525.32 * 100 - 100 = -.89% change in build time

standard deviation: 19.42 second

>MAKE=make at r187921 /usr/bin/time -h $MAKE buildworld >outfile 2>&1
>        1h48m7.30s real	   1h28m22.07s user     22m28.01s sys
>        1h48m0.94s real	   1h28m35.60s user     22m7.59s sys
>        1h48m4.20s real	   1h28m39.28s user     21m58.27s sys
>        (6487.30 + 6480.94 + 6484.20)/3 = 6484.15 sec ave
>        [10% improvement would reduce build by 10m48s]

standard deviation 3.18 seconds

>MAKE=make at r187921 /usr/bin/time -h $MAKE -s buildworld >outfile 2>&1
>        1h47m56.48s real   1h28m35.91s user     22m0.57s sys
>        1h47m55.65s real   1h28m40.66s user     21m56.70s sys
>        1h51m5.05s real	   1h28m48.22s user     22m21.96s sys
>        (6476.48 + 6475.65 + 6665.05)/3 = 6539.06 sec ave

    N           Min           Max        Median           Avg        Stddev
x   3       6475.65       6665.05       6476.48       6539.06     109.11133
+   3       6480.94        6487.3        6484.2     6484.1467     3.1803354
No difference proven at 95.0% confidence

>        If we toss out the high value and use 6476.48 twice, ave = 6476.20
>        => 6476.20 / 6539.06 * 100 - 100 = -.96% change in build time

We don't.  Statistics is not a television show.

>The NFS results are:
>
>MAKE=make at r187921 /usr/bin/time -h $MAKE -j16 buildworld >outfile 2>&1
>        35m33.59s real     1h33m20.08s user     52m1.04s sys
>        31m9.58s real      1h33m43.16s user     52m41.46s sys
>        31m18.94s real     1h33m40.45s user     52m41.58s sys
>        (2133.59 + 1869.58 + 1878.94)/3 = 1960.70 sec ave

standard deviation 149.79 seconds

>MAKE=make at r187921 /usr/bin/time -h $MAKE -j16 -Q -s buildworld >outfile 2>&1
>        31m42.62s real     1h33m28.64s user     52m4.32s sys
>        31m14.54s real     1h33m21.25s user     52m8.88s sys
>        31m17.26s real     1h33m22.48s user     52m10.93s sys
>        (1902.62 + 1874.54 + 1877.26)/3 = 1884.81 sec ave
>        => 1884.81 / 1960.70 * 100 - 100 = -3.87% change in build time

standard deviation 15.48 seconds

    N           Min           Max        Median           Avg        Stddev
x   3       1869.58       2133.59       1878.94     1960.7033     149.79737
+   3       1874.54       1902.62       1877.26     1884.8067     15.486631
No difference proven at 95.0% confidence

>        To be fair as in the local disk case, redoing the above:
>        (1878.94 + 1869.58 + 1878.94)/3 = 1875.82 sec ave (adjusted)
>        => 1884.81 / 1875.82 * 100 - 100 = .48% change in build time

That has nothing to do with "fair", that is fudging the numbers.

What you should have done, was realize that you need to run four
tests and throw the first one which primes the cache away.

>MAKE=make at r187921 /usr/bin/time -h $MAKE buildworld >outfile 2>&1
>        2h21m30.74s real   1h30m28.41s user     29m8.64s sys
>        2h21m2.15s real    1h30m25.54s user     29m2.99s sys
>        2h21m0.24s real    1h30m25.93s user     29m3.85s sys
>        (8490.74 + 8462.15 + 8460.24)/3 = 8471.04 sec ave

standard deviation 17.08 seconds

>MAKE=make at r187921 /usr/bin/time -h $MAKE -s buildworld >outfile 2>&1
>        2h20m48.71s real   1h30m26.02s user     29m7.66s sys
>        2h21m5.95s real    1h30m21.87s user     29m10.28s sys
>        2h21m7.70s real    1h30m29.85s user     29m4.29s sys
>        (8448.71 + 8465.95 + 8467.7)/3 = 8460.79 sec ave

standard deviation 10.49 seconds

No difference proven at 95.0% confidence

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


More information about the freebsd-arch mailing list