Benchmarks: AMD64 vs i386 on Dual 246 Opteron

[ray at redshift.com]

Freebsd-AMD64 list:

I recently completed benchmarking an evaluation server provided to us by our
hardware vendor in order to see if switching our cluster from Xeon based
machines to AMD based machines was worth while/cost effective

The machine provided was a Dual Opteron 246 using the Tyan S2881 motherboard.
It had 4GB or ram and included a single SATA hard drive.

I initially loaded FreeBSD 5.4 AMD64 on the machine, recompiled the kernel, etc.
and applied all the normal tweaks to apache, PHP, etc.  The machine, while
faster than our single 2.4 Ghz Xeon's, wasn't all that much faster (maybe only
10 to 15 percent).  

After speaking with AMD and doing further benchmarks, I was about to give up on
AMD and return the machine.  However, at the last minute, an engineer from AMD
suggested that perhaps loading the 32 bit version of FreeBSD (aka i386) might
improve performance, since it was possible that the overhead from 64 bit
pointers was causing the machine to run slower than expected.  He also explained
that the AMD should be running about 3 to 4 times faster than the single Xeon.

While this sounded like a long shot, I loaded FreeBSD 5.4 i386 on the machine
and after applying the exact same configuration to the OS, Apache, PHP and
MySQL, re-ran the benchmarks.  Much to my surprise, just changing the OS from 64
bit to 32 bit caused the machine to double in speed.  The results are attached
in an Excel spreadsheet.  So the exact same machine, running the identical
configuration, performed roughly twice as fast when running FreeBSD 5.4 i386 vs
FreeBSD 5.4 AMD64.  Something about this seems so wrong to me :-)

In speaking with one person off the list here, I was told that the FreeBSD AMD64
branch has actually been cleaned up substantially over the i386 code.  So
naturally I was expecting much better performance from a 64 bit machine running
the AMD64 code than the "older" i386 code.  I had also originally expected that
since this branch [the AMD64 branch] was specifically built around the AMD
CPU's, that it should run the best (and thus the fastest) on the AMD opteron
CPU's.  However, just the contrary turned out to be the case in benchmarking.

I'm wondering if anyone has any comments or thoughts on this?  The attached
benchmarks show transactions per second across localhost using Apache AB on the
same machine.  The first tests are with plain text files from 64 bytes to 50K in
size.  The next group is using a small and medium size PHP program.  The final
set relate to MySQL inserts, selects and updates.  As you can see from the data,
the exact same machine runs about twice as fast just by switching the OS from
AMD64 to i386.  But why?

The only answer I have so far as to why this may be the case is that perhaps
i386 uses 32 bit pointers which the CPU(s) can handle faster (thus less overhead
for the CPU).  But it still seems odd to me that if FreeBSD AMD64 is written
specifically for the 64 bit CPU, why doesn't the machine perform better when
running it?

I'm also wondering if there is a compiler switch on AMD64 that could be used
(perhaps in /etc/make.conf or something) that would force the AMD64 version to
run in 32 bit mode only - since that would be an interesting comparison as well.

I welcome any/all comments, suggestions, insight. 

Thanks.

Ray






-------------- next part --------------
A non-text attachment was scrubbed...
Name: bench1.xls
Type: application/octet-stream
Size: 13824 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-amd64/attachments/20050728/5de179ab/bench1.obj