Performance research

[Kris Kennaway]

On Thu, Apr 24, 2008 at 09:58:22PM +0200, David Naylor wrote:
> Hi,
> 
> (If you want to skip to the questions (since this is a questions mailing 
> list :-) please see the 2nd last paragraph)
> 
> I am interested in the performance difference of using either -Os or -O2.  
> From my preliminary research I have found that in most cases -Os 
> outperforms -O2 (and produces smalled binaries).  
> 
> However this is only in a very isolated situation and does not make any case 
> for a system wide change therefore I would like to run some experiments to 
> measure the difference between the two flags.  
> 
> I plan to use a live CD (with a system using 3GB) and having both the source 
> and object tree in memory (so as to avoid the harddrive influencing the 
> times).  Since the CD is very small I am expecting caching to eliminate any 
> influence of the slow CD read times of any of the files required to build 
> world.  
> 
> The tests will be automated by scripts with two CD's, the first with -Os and 
> the second with -O2 binaries for the kernel and world (both amd64).  I am 
> running an Intel Core 2 Duo 2.2 and 3GB of ram (as mentioned before).  I am 
> hoping that only the execution speed will be measured without any other 
> factors influencing (such a lack of memory [and thus -Os having the 
> advantage]) the results.  
> 
> Onto the questions:
> 1) Is there any other (better) tool I should use to measure the performance of 
> building world (other than time)?

Are you really interested in benchmarking how fast gcc is, i.e. is
your workload really "compile lots of code"?  You'll be mostly
measuring the speed of gcc when doing varying amounts of optimization
work, which will be orders of magnitude more of an effect than the
difference between -Os and -O2 in the gcc binary itself.  You will
have to be careful to avoid this, if it's not your intention.

> 2) Since (I think) all of FreeBSD sources of aliassing safe (I have been 
> running a system without -fno-strict-aliassing and optimised for a while 
> without a problem) should I leave it out of the build flags (i.e. 
> CFLAGS=-Os -pipe)
> 3) Any other suggestions on refining this experiment?
> 4) Should this experiment produce potential, or conclusive, results any other 
> suggestions of experiments to run (perhaps a database or other flags?) 

Optimization levels only have a measurable effect when you are running
a highly CPU bound job, so choose a workload appropriately and measure
carefully (tip: use ministat to decide whether your numbers show any
significant effect).

Kris

--
In God we Trust -- all others must submit an X.509 certificate.
    -- Charles Forsythe <forsythe at alum.mit.edu>