Continual benchmarking / regression testing?

[Alan Somers]

On Wed, Jan 8, 2014 at 1:13 PM, Erik Cederstrand
<erik+lists at cederstrand.dk> wrote:
> Den 08/01/2014 kl. 17.38 skrev Alan Somers <asomers at FreeBSD.org>:
>>
>> I like that you stored test results in a SQL database.  My SQL-foo is
>> poor, so ATM my framework is using CSV.  It also looks like you've got
>> code to generate a website.  Do you have any example output?
>
> Yes, there’s a website to filter results, generate graphs, see commit messages between two data points and show the hardware and software configuration of the client running the benchmark. A continuous benchmarking framework is only useful if it can assist you in analyzing the data, finding regressions and their cause.

I meant, do you have any generated html that you can share?

>
>> The PXE stuff, however, does not belong in the
>> benchmark framework, IMHO.  I think that the benchmark framework
>> should just include the benchmarking and system profiling aspect, not
>> system configuration.  Provisioning and configuring systems can be
>> done in a separate utility, one that can be shared, for example, with
>> the continuous Kyua tester.
>
> System configuration affects benchmark results, so that needs to be recorded along with the benchmark results. My work was intended as a complete continuous benchmarking system with a build machine that produces OS installation images and tells clients what to install and what to run. But I agree that a benchmark framework contains many self-contained parts that could be shared among projects.

My strategy was to separate system configuration from system analysis.
 That way, the user can configure the system however he likes, using
any tools.  Then the framework will analyze the system in as much
detail as possible.  It will determine the CPU type, CPU count, memory
type, memory amount, kernel version, network interface configuration,
filesystem, filesystem properties, zpool topology, hard disk model,
etc.  The analysis engine is a lot of work, but its more robust and
flexible than tying system configuration into the framework.

This separation of analysis from configuration allows the
configuration aspect to be handled by a separate tool, that knows
nothing of benchmarks.  Sequencing the benchmarks can then be run by a
short sh script or something.

-Alan

>
> Erik