(Another) simple benchmark

[Ivan Voras]

Sorry, I forgot about performance@ mailing list, moving the discussion 
there - please use performance at freebsd.org or equivalent address when 
replying (and drop current@).

Steve Hodgson wrote:
> Ivan Voras wrote:
> 
>>Today I had the opportunity to experiment for a short time with a
>>4-CPU Xeon machine with hyperthreading (-> 8 logical CPU-s) and 3GB
>>RAM. In absence of anything smarter to do, I installed WBEL 3 Linux
>>(non-RedHat version of RedHat Enterprise Linux) which was accidentally
>>on the table and after toying around I've run a simple benchmark - ab
>>(apachebench, included by default with apache server) over localhost
>>to a local apache2 server, on a static home-page file. The command
>>line was "ab -n 100000 -c 100 http://localhost/". Since I still had
>>time with the machine, I then downloaded and installed FreeBSD 6.1 to
>>try to beat the score. Unfortunately, not only I couldn't beat the
>>score, but the results were *extremely* bad. Since I'm still not sure
>>it isn't my fault, I won't post the actual results, but I wonder...
>>
>>Apache is a well known server-grade product, which doesn't use
>>threading (it was preforked in both systems), doesn't call
>>gettimeofday() constantly, uses sendfile(), and in short, is very
>>different from MySQL. It shouldn't behave this badly on FreeBSD.
>>
>>Some observations from the benchmark runs:
>>- Linux was maxed out with ~50% total idle time, each logical
>>processor (hyperthreading was enabled) had ~25% user + ~25% sys time.
>>It still delivered order-of-magnitude better results. WBEL3 uses Linux
>>2.4.x kernel (i.e. old kernel)
>>- FreeBSD CPU time was 100% spent, with 90%-95% spent in sys time
>>- On FreeBSD, enabling hyperthreading got me a slowdown of *4x* over
>>the initial (bad) results (some slowdown is not surprising, 4x
>>slowdown is).
>>- Linux "load average" never went above 8, FreBSD's went > 60; this
>>could be just a difference in accounting, or it could not - I don't know.
>>
>>What is needed to reproduce this simple benchmark:
>>- SMP machine, the more CPUs the better
>>- installed software: only apache20. The "ab" benchmark is included in
>>it. Leave apache's default configuration as-is (i.e. preforked, max
>>150 parallel clients allowed)
>>- A simple static HTML page (Apache's default "welcome, but this site
>>is not configured" should be ok).
>>- my invocation of "ab" was with "-n 100000 -c 100" ; the (n)umber of
>>requests could be modified to fit local CPU speed, but (c)oncurrency
>>shouldn't (100 parallel requests is not uncommon).
>>
> 
> Apache may not use gettimeofday, but ab certainly does. here is a truss
> output of ab:...

I don't think this would result in 90% time being spent in sys with 
large load averages.

> A fairer test would be to use ab on a second box and keep ab on the same
> OS - you're changing two things at once here. You're also at the mercy
> of how the scheduler places the processes across the different CPU's
> (since there is one ab process and multiple httpd processes). Why was
> Linux stuck at 50% utilization per CPU (surely it should reach 100% with
> no disk or network constraints)? Perhaps it is sleeping whilst FreeBSD
> spins?

> If you are interested in performance try Lighttpd - that is what people
> use if they want high transaction rates (x10 higher in tests I've done
> on Linux) - and it is a single process so scheduling isn't so significant.

Using lighttpd or ab on other host is not really applicable - I wasn't 
trying to configure the box for production, only ran the benchmarks out 
of curiosity. Apache *should* work ok out of the box on FreeBSD :)
(btw. apache was installed from binary package)

Here are the apache config files: 
http://ivoras.sharanet.org/stuff/httpd.conf.tbz