Continual benchmarking / regression testing?

[Julio Merino]

On Tue, Jan 7, 2014 at 4:17 PM, Ivan Voras <ivoras at freebsd.org> wrote:
> On 7 January 2014 17:11, Julio Merino <julio at meroh.net> wrote:
>> On Tue, Jan 7, 2014 at 4:09 PM, Ivan Voras <ivoras at freebsd.org> wrote:
>>> Hello,
>>>
>>> Is someone working on a contitual benchmarking / regression testing
>>> project for FreeBSD? I seem to recall there was a post several months
>>> ago but I can't find it.
>>
>> See http://wiki.freebsd.org/TestSuite for the current efforts.
>
> Ok, by looking at the wiki page and http://kyua1.nyi.freebsd.org/
> (only stable/10 is available btw), it looks like this effort is
> centered on correctness, not performance benchmarking?

Correct.  I have basically no thoughts at the moment on benchmarking,
but that's certainly something worth tackling.

My email crossed Alan's, but here go my thoughts anyway. I feel
integrating some basic kind of performance testing into the test suite
might be beneficial if only to catch regressions, but detailed
performance testing may be difficult for all the reasons mentioned by
Alan.

A simple possibility could be to explicitly mark specific tests as
"benchmark tests" so that Kyua could measure and record their run
time.  In fact, Kyua already records the run time of tests and
maintains historical data.  What would be missing is a way to graph
the results and to alert when the measurements differ above some
thresholds.

But then, benchmarking tests will have special requirements --
particularly during the setup of dependencies, the setup of the
machine (to ensure there is no background noise) and also due to the
large amount of tunables that may be involved.  Plugging such tests
into a correctness test suite, except for simple tests, is hard and
may be not such a great idea.

> Is Kyua easily
> adaptable to include graphs and other more visually attractive
> presentation types?

Not at the moment, but better reporting is the thing I want to tackle
the soonest. See the planning details
(http://julipedia.meroh.net/2014/01/freebsd-test-suite-goals-and-planning.html)
for some more information.  In particular, visit the "test matrix"
sheet of the planning spreadsheet.  But again, there is nothing there
regarding performance testing.

-- 
Julio Merino / @jmmv