Continual benchmarking / regression testing?

[Erik Cederstrand]

Den 08/01/2014 kl. 17.38 skrev Alan Somers <asomers at FreeBSD.org>:
> 
> I like that you stored test results in a SQL database.  My SQL-foo is
> poor, so ATM my framework is using CSV.  It also looks like you've got
> code to generate a website.  Do you have any example output?

Yes, there’s a website to filter results, generate graphs, see commit messages between two data points and show the hardware and software configuration of the client running the benchmark. A continuous benchmarking framework is only useful if it can assist you in analyzing the data, finding regressions and their cause.

> The PXE stuff, however, does not belong in the
> benchmark framework, IMHO.  I think that the benchmark framework
> should just include the benchmarking and system profiling aspect, not
> system configuration.  Provisioning and configuring systems can be
> done in a separate utility, one that can be shared, for example, with
> the continuous Kyua tester.

System configuration affects benchmark results, so that needs to be recorded along with the benchmark results. My work was intended as a complete continuous benchmarking system with a build machine that produces OS installation images and tells clients what to install and what to run. But I agree that a benchmark framework contains many self-contained parts that could be shared among projects.

Erik