Possible Threading problem with -CURRENT / MySQL?

[Robert Watson]

On Tue, 15 Jun 2004, Julian Elischer wrote:

> On Mon, 14 Jun 2004, mike wrote:
> 
> > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x
> > as well. it sounds like scheduler/threading is to blame but we were not
> > able to give sufficient or proper motivation to the folks who could
> > examine this deeper - we even offered $500 cash to whomever stepped up to
> > help resolve this.
> > 
> > linux runs almost 2x as fast on the same hardware with no configuring -
> > and we get nearly the same results running in single CPU mode vs. dual CPU
> > mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> > combination.
>  
> You complained about this some time ago and you have still not responded 
> with the information I suggest..

I sent this to Jeremy privately, since it was just some preliminary
measurements, but figured I'd send it publically since the results were
interesting (if tentative, I need to do a lot more work to make them
useful.  There are a number of variables I need to look at including:

- Disabling HTT.  A chat with Scott Long this evening suggests that HTT
  may be substantially hurting the test cases given increased IPIs, etc.
  Unfortunately, it looks like I can't easily twiddle HTT without being
  local to the machine, and I'm at home right now (it being 1:30am and
  all).   Removing HTT may help substantially with the dip in performance
  in the SMP configuration.

- I'd like to compare against RELENG_4 and a recent Linux kernel.
  Unfortunately, the box is configured for neither right now.

- I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd
  like to try SCHED_4BSD.

- This was without adaptive mutexes, which seem to be helpful for others,
  so I should give them a try.

I don't have any amd64 hardware, so I don't know what if any role it will
play in the results.  The performance drop observed in the report appears
to be on amd64 (I may have misread).

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Senior Research Scientist, McAfee Research


----
Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT)
From: Robert Watson <rwatson at FreeBSD.org>
To: JG <amd64list at jpgsworld.com>
Subject: Re: Possible Threading problem with -CURRENT / MySQL?


On Tue, 15 Jun 2004, JG wrote:

> Fwiw, it has to be something that was committed between May 18th and
> yesterday.  ~May 18th was the last time I built -CURRENT during my last
> round of testing and I did not have any of these problems.  Then someone
> emailed me recently and said there were some commits that might effect
> the outcome of the mysql benchmarks. 

Ok, so these results are on a dual-processor XEON + hyperthreads, so four
logical processors.  I used two dates off CVS, 20040515 and 20040615.  I
also benchmarked my netperf branch.  I don't have RELENG_4 on the box, but
might be able to load RELENG_4 on it later this week.  In each case, I
took ten samples, dropped the first value as getting into the cache, and
took the mean of the rest.  For this test, I used the select test; I'll
try the other smack query set tomorrow.  In each case, I ran with "10
1000" as the arguments to the test.  I used the default threading
configuration in -CURRENT, which is libpthread (libkse). 

                                Mean       Stdev
20040515-UP                     4752.27    14.63
20040515-SMP                    2550.35    19.23

20040615-UP                     4898.71    22.39
20040615-SMP                    2666.93    32.01

Netperf-UP-giant                4902.41    14.3
Netperf-SMP-giant               2566.18    16.83
Netperf-UP-mpsafe               4799.35    22.04
Netperf-SMP-mpsafe              3022.51    18.06

Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages
the SMP numbers a fair amount due to additional IPIs without benefit. 
However, the numbers basically suggest that on my hardware, the UP
configuration is marginally faster than it was last month, and that if you
throw in the netperf branch, the SMP case is a moderate amount faster. 
This suggests that either I'm just lucky, or that the performance loss
might be specific to the amd64 version of FreeBSD.  I'm going to run some
more numbers tomorrow and try to post something more rigorous to the
-threads list. 

I don't have RELENG_4 on the box or Linux on the box, but I may get a
chance to later this week. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Senior Research Scientist, McAfee Research