Possible Threading problem with -CURRENT / MySQL?
Robert Watson
rwatson at freebsd.org
Thu Jun 17 05:34:11 GMT 2004
On Wed, 16 Jun 2004, Julian Elischer wrote:
> Robert, can you try see if HZ=5000 (up from 100) changes the
> performance?
I haven't explored the HZ=BIGGER case very thoroughly, but ran a couple of
test cases with HZ at 1000 instead of 100, and found that generally it
resulted in a slight (1.5%) performance drop in the smattering of cases I
looked at. The cases I looked at were NETPERF-SMP-MPSAFE-ADMTX-HZ=1000,
NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT-HZ=1000.
In general, I found that the biggest variables improving performance
relative to out-of-box SMP configuration were:
- Removing Giant from UNIX domain sockets
- Running with SCHED_4BSD instead of SCHED_ULE
- Disabling HTT
- Using ADAPTIVE_MUTEXES
By doing these things, I went from an out-of-box queries/sec for the
simple select "smack" with 11 clients from 2667 q/s to 6955 q/s, or a bit
over doubling the transaction rate. That as compared to 4939 q/s, which
was the best UP result I got (Giant-free UNIX domain sockets, SCHED_4BSD):
20040615-UP 4898.71 q/s (22.39 stddev)
20040615-SMP 2666.93 q/s (32.01 stddev)
NETPERF-UP-MPSAFE-4BSD 4939.89 q/s (74.06 stddev)
NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT 6955.18 q/s (156.91 stddev)
Some of my 4BSD performance numbers were also slightly pessimized with
what appears to be a property of the 4BSD scheduling period being longer
than ULE. With ULE, the results settled on one loop of the benchmark.
With 4BSD, it sometimes took 2-3 loops of the benchmark to settle. As a
result, 4BSD stddev's are also generally higher. I haven't tried
regenerating the results dropping those outliers.
Using out-of-box -CURRENT without the netperf patches necessary to run
UNIX domain sockets safely, I got best results using:
20040615-UP-4BSD 4886.84 q/s (46.03 stddev)
20040615-SMP-ADMTX-4BSD-!HTT 5838.76 q/s (45.11 stddev)
In my environment, I couldn't easily test ULE without HTT, since the HTT
disabling sysctl/tunable with ULE results in a hang.
(Obviously, in all of the above, WITNESS and INVARIANTS are disabled; I
didn't disable the userspace malloc debugging flags).
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org Senior Research Scientist, McAfee Research
>
>
> On Wed, 16 Jun 2004, Robert Watson wrote:
>
> >
> > On Tue, 15 Jun 2004, Julian Elischer wrote:
> >
> > > On Mon, 14 Jun 2004, mike wrote:
> > >
> > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x
> > > > as well. it sounds like scheduler/threading is to blame but we were not
> > > > able to give sufficient or proper motivation to the folks who could
> > > > examine this deeper - we even offered $500 cash to whomever stepped up to
> > > > help resolve this.
> > > >
> > > > linux runs almost 2x as fast on the same hardware with no configuring -
> > > > and we get nearly the same results running in single CPU mode vs. dual CPU
> > > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> > > > combination.
> > >
> > > You complained about this some time ago and you have still not responded
> > > with the information I suggest..
> >
> > I sent this to Jeremy privately, since it was just some preliminary
> > measurements, but figured I'd send it publically since the results were
> > interesting (if tentative, I need to do a lot more work to make them
> > useful. There are a number of variables I need to look at including:
> >
> > - Disabling HTT. A chat with Scott Long this evening suggests that HTT
> > may be substantially hurting the test cases given increased IPIs, etc.
> > Unfortunately, it looks like I can't easily twiddle HTT without being
> > local to the machine, and I'm at home right now (it being 1:30am and
> > all). Removing HTT may help substantially with the dip in performance
> > in the SMP configuration.
> >
> > - I'd like to compare against RELENG_4 and a recent Linux kernel.
> > Unfortunately, the box is configured for neither right now.
> >
> > - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd
> > like to try SCHED_4BSD.
> >
> > - This was without adaptive mutexes, which seem to be helpful for others,
> > so I should give them a try.
> >
> > I don't have any amd64 hardware, so I don't know what if any role it will
> > play in the results. The performance drop observed in the report appears
> > to be on amd64 (I may have misread).
> >
> > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
> > robert at fledge.watson.org Senior Research Scientist, McAfee Research
> >
> >
> > ----
> > Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT)
> > From: Robert Watson <rwatson at FreeBSD.org>
> > To: JG <amd64list at jpgsworld.com>
> > Subject: Re: Possible Threading problem with -CURRENT / MySQL?
> >
> >
> > On Tue, 15 Jun 2004, JG wrote:
> >
> > > Fwiw, it has to be something that was committed between May 18th and
> > > yesterday. ~May 18th was the last time I built -CURRENT during my last
> > > round of testing and I did not have any of these problems. Then someone
> > > emailed me recently and said there were some commits that might effect
> > > the outcome of the mysql benchmarks.
> >
> > Ok, so these results are on a dual-processor XEON + hyperthreads, so four
> > logical processors. I used two dates off CVS, 20040515 and 20040615. I
> > also benchmarked my netperf branch. I don't have RELENG_4 on the box, but
> > might be able to load RELENG_4 on it later this week. In each case, I
> > took ten samples, dropped the first value as getting into the cache, and
> > took the mean of the rest. For this test, I used the select test; I'll
> > try the other smack query set tomorrow. In each case, I ran with "10
> > 1000" as the arguments to the test. I used the default threading
> > configuration in -CURRENT, which is libpthread (libkse).
> >
> > Mean Stdev
> > 20040515-UP 4752.27 14.63
> > 20040515-SMP 2550.35 19.23
> >
> > 20040615-UP 4898.71 22.39
> > 20040615-SMP 2666.93 32.01
> >
> > Netperf-UP-giant 4902.41 14.3
> > Netperf-SMP-giant 2566.18 16.83
> > Netperf-UP-mpsafe 4799.35 22.04
> > Netperf-SMP-mpsafe 3022.51 18.06
> >
> > Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages
> > the SMP numbers a fair amount due to additional IPIs without benefit.
> > However, the numbers basically suggest that on my hardware, the UP
> > configuration is marginally faster than it was last month, and that if you
> > throw in the netperf branch, the SMP case is a moderate amount faster.
> > This suggests that either I'm just lucky, or that the performance loss
> > might be specific to the amd64 version of FreeBSD. I'm going to run some
> > more numbers tomorrow and try to post something more rigorous to the
> > -threads list.
> >
> > I don't have RELENG_4 on the box or Linux on the box, but I may get a
> > chance to later this week.
> >
> > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
> > robert at fledge.watson.org Senior Research Scientist, McAfee Research
> >
> >
> >
>
>
More information about the freebsd-current
mailing list