FreeBSD MySQL still WAY slower than Linux

[Robert Watson]

On Fri, 17 Jun 2005, David Sze wrote:

>        FreeBSD/amd64 5.4-RELEASE (libpthread, system and process scope)
>        FreeBSD/amd64 6.0-CURRENT (libpthread, libthr, system and process 
> scope)
>        CentOS/amd64 4.0 (i.e. RHEL4.0)
>
> I couldn't get libthr to work on FreeBSD/amd64 5.4-RELEASE, mysqld would 
> immediately coredump and the stacktrace looked like it was corrupted 
> (i.e. hundreds of stack frames, all of which were ???).

The libthr in 5.x is basically an early prototype compared to the more 
mature work on 6.x, so this result is not all that surprising.  David Xu 
has put a very large amount of work into the libthr on 6.x with some very 
good results.

> It turns out that the problem was the same thing everyone usually points 
> the finger at, but no one actually mentioned this time:  Linux mounts 
> its partitions async by default.  I don't have the exact numbers in 
> front of me right now, but these were the ballpark figures (I'm not 
> going to separate out results for all of the different threading 
> libraries for FreeBSD because the deltas weren't huge):

This is actually an interesting observation -- on 6.x using P4 Xeon 
hardware, I've seen a substantial performance improvement from running 
with libthr with MySQL.  Sometimes as much as 30% - 50%.  However, amd64 
is quite a different beast.

BTW, could you run 'file' on the Linux and FreeBSD MySQL binaries and 
confirm that they're both either 32-bit i386 binaries, or 64-bit amd64 
binaries?

> That last CentOS number is not a typo, it was an order of magnitude 
> slower. I didn't try other file systems on CentOS, just the default 
> ext3.  It's possible that reiserfs or xfs might not be as affected by 
> switching from async to sync.
>
> So my production server is now happily running mysql 4.1 on 6.0-CURRENT 
> :).

I was interested to observe, in some benchmarking a few months ago, that 
4.0 MySQL performance on FreeBSD 6.x is a *lot* faster than 4.1 MySQL 
performance.  I don't track MySQL feature development, but I couldn't help 
but wonder if either there had been substantial reliability changes that 
impacted file system semantics (which might be indicated by sync/async 
issues), or whether there had been a lot of performance optimization work 
for Linux that had swung its use of IPC/etc primitives towards ones that 
work better on Linux than FreeBSD?

BTW, could you confirm that on 6.x, you're setting /etc/malloc.conf so 
that it's not scrubbing memory on free()?  In particular:

   ln -s jz /etc/malloc.conf

as root.  Most 6.x debugging features are set as part of the kernel 
configuration, and your performance results suggest that you've caught 
them all already, I think, but this one is user space.  Also, make sure 
you are compiling without INVARIANT_SUPPORT -- for packet send tests, I 
find that even without INVARIANTS, I see a 4% hit for having 
INVARIANT_SUPPORT in the kernel.

Robert N M Watson