Automated performance testing

[Robert Watson]

On Sun, 30 Jan 2005, Jim C. Nasby wrote:

> With all the discussion of performance testing between 4.11, 5.3, and
> Linux, would it be useful to make performance testing part of the
> automated testing that already occurs (via tinderbox, iirc). Doing so
> might make it easier to detect performance impacting changes, as well as
> making performance testing easier in general. 

Yes, it would be quite valuable.  I've been hoping to set up something
like this for a while, but have never found the opportunity.  I have been
tracking the long term behavior of MySQL performance as part of the
netperf work, but because testing is fairly hardware and time consuming,
the polling intervals are uneven, and not quite close enough to nail down
culprits.  I'd really like to see a small and fairly well-defined set of
tests run every couple of days so we can show long term graphs, and catch
regressions quickly.  Unfortunately, this is a bit harder than
tinder-boxing, because it involves swapping out whole system
configurations, recovering from the inevitable failure modes, etc, which
proves to be the usual sticking point in implementing this.  However, I'd
love to see someone work on it :-).

Robert N M Watson