Continual benchmarking / regression testing?

[Alan Somers]

On Tue, Jan 7, 2014 at 11:07 AM, Julio Merino <julio at meroh.net> wrote:
> On Tue, Jan 7, 2014 at 4:17 PM, Ivan Voras <ivoras at freebsd.org> wrote:
>> On 7 January 2014 17:11, Julio Merino <julio at meroh.net> wrote:
>>> On Tue, Jan 7, 2014 at 4:09 PM, Ivan Voras <ivoras at freebsd.org> wrote:
>>>> Hello,
>>>>
>>>> Is someone working on a contitual benchmarking / regression testing
>>>> project for FreeBSD? I seem to recall there was a post several months
>>>> ago but I can't find it.
>>>
>>> See http://wiki.freebsd.org/TestSuite for the current efforts.
>>
>> Ok, by looking at the wiki page and http://kyua1.nyi.freebsd.org/
>> (only stable/10 is available btw), it looks like this effort is
>> centered on correctness, not performance benchmarking?
>
> Correct.  I have basically no thoughts at the moment on benchmarking,
> but that's certainly something worth tackling.
>
> My email crossed Alan's, but here go my thoughts anyway. I feel
> integrating some basic kind of performance testing into the test suite
> might be beneficial if only to catch regressions, but detailed
> performance testing may be difficult for all the reasons mentioned by
> Alan.

For a good example of performance regression testing, see the Linux
kernel tracker at http://www.phoromatic.com/kernel-tracker.php? .  As
you can see, the hardware and software profiling is weak, and
regression detection is very inexact.  It wouldn't be possible to
distill a single benchmark's results into a "regressed"/"didn't
regress" result.  Accurately detecting regressions would probably take
some moderately sophisticated statistics.

>
> A simple possibility could be to explicitly mark specific tests as
> "benchmark tests" so that Kyua could measure and record their run
> time.  In fact, Kyua already records the run time of tests and
> maintains historical data.  What would be missing is a way to graph
> the results and to alert when the measurements differ above some
> thresholds.
>
> But then, benchmarking tests will have special requirements --
> particularly during the setup of dependencies, the setup of the
> machine (to ensure there is no background noise) and also due to the
> large amount of tunables that may be involved.  Plugging such tests
> into a correctness test suite, except for simple tests, is hard and
> may be not such a great idea.

Yeah, that's basically the same conclusion I came to.

>
>> Is Kyua easily
>> adaptable to include graphs and other more visually attractive
>> presentation types?
>
> Not at the moment, but better reporting is the thing I want to tackle
> the soonest. See the planning details
> (http://julipedia.meroh.net/2014/01/freebsd-test-suite-goals-and-planning.html)
> for some more information.  In particular, visit the "test matrix"
> sheet of the planning spreadsheet.  But again, there is nothing there
> regarding performance testing.

So is Julipedia now the best source for Kyua news instead of
http://engineering-kyua.blogspot.com/ ?  I have the latter in my RSS
feed, but it hasn't updated for awhile.

-Alan

>
> --
> Julio Merino / @jmmv
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"