ATF config variables for FreeBSD tests

[Alan Somers]

On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino <jmmv at freebsd.org> wrote:
> On Sat, Mar 15, 2014 at 7:04 AM, Alan Somers <asomers at freebsd.org> wrote:
>> Another thread recently raised the issue of tests that cause
>> potentially harmful changes to sysctl variables.  We all agreed to
>> conditionalize that test based on an ATF config variable.  Now Ken and
>> I both want to add tests with different harmful side effects.  I
>> propose the following list of config variables.  If they sound good to
>> everyone, I'll add them to tests(7) and to an example kyua.conf file
>> in devel/kyua.
>>
>> test_suites.FreeBSD.allow_sysctl_side_effects
>> If defined, enables tests that cause potentially harmful changes to
>> globally relevant sysctl variables.  Requested by Peter for
>> kern_descrip_test:maxfiles__increase
>>
>> test_suites.FreeBSD.fibs
>> Defined to a space delimited list of fib numbers.  Tests that change
>> routing tables must use these fibs instead of the default.  They will
>> be skipped if the variable is undefined.  I'm working on such tests
>> right now.
>>
>> test_suites.FreeBSD.disks
>> Defined to a space delimited list of disk device nodes.  Tests that
>> format, mount, write to, disable, or have any other harmful effect on
>> disks must use these devices.
>>
>> test_suites.FreeBSD.allow_devfs_side_effects.
>> If defined, enables tests that can create or destroy semipermanent
>> device nodes, like disk devices.  Without this variable, tests may
>> still create and destroy device nodes that are normally transient,
>> like /dev/tap* and /dev/pts/*.  This variable requires further
>> explanation:
> [...]
>
> I think these all sound reasonable.
>
> Can these expected side-effects be reversed in the test cleanup?

For the first two, yes.  I doubt that Kyua could do it automatically,
but each test case certainly can.  Perhaps even for the fourth, though
it would be tricky.  But there's no way to reverse the effects for the
third.  The ZFS tests, for example, create and destroy zpools.  How
would you reverse that?  You can't.  Whatever was previously on the
disk is gone.

>
> The descriptions above should clearly mention these details. For
> example, for sysctl-related tests I'd expect them to restore the
> original values just fine... but maybe that's not the case in the
> other examples you mention?
>
>> test_suites.FreeBSD.unit_tests
>> test_suites.FreeBSD.functional_tests
>> test_suites.FreeBSD.stress_tests
>> These three variables would address a different concern of Peter's.
>> He was concerned that his test might require too much run time,
>> annoying other users of the rest of the test suite.  By using these
>> variables, a sysadmin could easily disable stress tests, which tend to
>> take a very long time to run.  Conversely, unit tests are normally
>> very fast.  A kyua run that only ran unit tests could easily be done
>> after every buildworld, for example, while one that ran functional
>> tests as well could be done perhaps several times per day.  This is
>> just an idea I have.  I haven't tried it in practice.
>>
>> What do you think?
>
> Doing this separation by sizes is important for the reasons you
> outline, but I have a different proposal that I have had in mind for a
> while and only recently started to work on.
>
> The idea is, simply, to introduce a "size" metadata variable to the
> test cases and then make every test declare if they are "small",
> "medium", "large", etc. With this, the user is allowed to customize,
> in kyua.conf, the timeouts for every test case size -- with some
> reasonable defaults for modern machines. (I've seen this implemented
> in other test suites and is an approach that works pretty well.)
>
> The reason to do it this way is to kill the "timeout" property, which
> is unreliable as it is not possible to make good predictions on run
> time when you are going to run the same test on multiple
> architectures. With this approach, we'd then say that qemu VMs have
> much longer timeouts for every class, for example.

This sounds good, as long as test authors have good guidelines on how
to appraise their tests' sizes.

>
> And lastly, we'd just need a simple "filtering" feature in the kyua
> cli to allow specifying which size of tests to run (or to filter by
> any other metadata property, for that matter).

I don't like this idea.  We tried it at work, and it didn't work out
very well.  Basically, I automatically assigned sizes (short, medium,
long) to all of our tests based on their runtimes, so users could
select to only run the short or medium tests on the bench.  The
problem is that the classification didn't make sense.  For an
expedited test run, you don't want the shortest tests; what you want
are the tests with the most value per unit time.  Unit tests have a
high value and a very short runtime.  Stress tests have a very long
runtime and arguably low value since they don't always have consistent
results.

At my previous job there was a large department whose duties included
curating test suites and deciding which tests would be included in the
short runs.  FreeBSD isn't going to have that, but we could still ask
test authors to classify tests along the lines that I suggested.

-Alan