Automated performance testing

[Eric Anderson]

Robert Watson wrote:
> On Sun, 30 Jan 2005, Jim C. Nasby wrote:
> 
> 
>>With all the discussion of performance testing between 4.11, 5.3, and
>>Linux, would it be useful to make performance testing part of the
>>automated testing that already occurs (via tinderbox, iirc). Doing so
>>might make it easier to detect performance impacting changes, as well as
>>making performance testing easier in general. 
> 
> 
> Yes, it would be quite valuable.  I've been hoping to set up something
> like this for a while, but have never found the opportunity.  I have been
> tracking the long term behavior of MySQL performance as part of the
> netperf work, but because testing is fairly hardware and time consuming,
> the polling intervals are uneven, and not quite close enough to nail down
> culprits.  I'd really like to see a small and fairly well-defined set of
> tests run every couple of days so we can show long term graphs, and catch
> regressions quickly.  Unfortunately, this is a bit harder than
> tinder-boxing, because it involves swapping out whole system
> configurations, recovering from the inevitable failure modes, etc, which
> proves to be the usual sticking point in implementing this.  However, I'd
> love to see someone work on it :-).

Maybe it would help to come up with a list of tests, or even just things to be tested initially.  Then one could get a sense of what tests need to be run, and how to do it..  I wonder if it's worth building a little app that runs a suite of other included apps (found in the ports collection), grabs system hardware and configuration information, runs the tests, and sends them to a central location.  Then we can have OS, hardware, software config, etc information, and compare system vs system, OS ver, etc, and people just build the fbsdperftest port (which installs a slew of other ports, like iozone, bonnie, apache (HTTP testing), mysql (DB testing?), etc), then they run the program, it gathers the data, asks a couple of questions, and sends the data off to a central spot for analyzing later.  Then, one could set up a machine that just does the cvsup, buildworld, ..., fbsdperftest, repeat..  

Eric



-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
I have seen the future and it is just like the present, only longer.
------------------------------------------------------------------------