ATF config variables for FreeBSD tests

[Julio Merino]

On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers <asomers at freebsd.org> wrote:
> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino <jmmv at freebsd.org> wrote:
[...]
>> I think these all sound reasonable.
>>
>> Can these expected side-effects be reversed in the test cleanup?
>
> For the first two, yes.  I doubt that Kyua could do it automatically,
> but each test case certainly can.

Right. I do not think Kyua should get into these tricky details
either, if only because they are too OS-specific. But we can certainly
offer additional scripts/functions (where it makes sense) in the
FreeBSD test suite to simplify the test cases that might need this.

>  Perhaps even for the fourth, though
> it would be tricky.  But there's no way to reverse the effects for the
> third.  The ZFS tests, for example, create and destroy zpools.  How
> would you reverse that?  You can't.  Whatever was previously on the
> disk is gone.

Ah, I guess I missed that detail.

Couldn't those tests run on top of vnd devices though?

[...]
>> And lastly, we'd just need a simple "filtering" feature in the kyua
>> cli to allow specifying which size of tests to run (or to filter by
>> any other metadata property, for that matter).
>
> I don't like this idea.

I hope you are objecting to the filtering by test sizes, not the
filtering itself!  See below.

>  We tried it at work, and it didn't work out
> very well.  Basically, I automatically assigned sizes (short, medium,
> long) to all of our tests based on their runtimes, so users could
> select to only run the short or medium tests on the bench.  The
> problem is that the classification didn't make sense.  For an
> expedited test run, you don't want the shortest tests; what you want
> are the tests with the most value per unit time.  Unit tests have a
> high value and a very short runtime.  Stress tests have a very long
> runtime and arguably low value since they don't always have consistent
> results.
>
> At my previous job there was a large department whose duties included
> curating test suites and deciding which tests would be included in the
> short runs.  FreeBSD isn't going to have that, but we could still ask
> test authors to classify tests along the lines that I suggested.

That's a good point. If I understand you correctly, what is important
is to manually curate the set of "smoke tests" that run very quickly
and that provide a reasonably high assurance that major subsystems
haven't broken.

We can do this too pretty easily, but configuration variables are the
wrong mechanism. We can already define this information in metadata
properties as long as you prefix them with X-. So all we'd need is a
way to tell Kyua to only run tests marked with such property (i.e.
with the filtering feature described above).