ATF config variables for FreeBSD tests

[Julio Merino]

On Sat, Mar 15, 2014 at 7:04 AM, Alan Somers <asomers at freebsd.org> wrote:
> Another thread recently raised the issue of tests that cause
> potentially harmful changes to sysctl variables.  We all agreed to
> conditionalize that test based on an ATF config variable.  Now Ken and
> I both want to add tests with different harmful side effects.  I
> propose the following list of config variables.  If they sound good to
> everyone, I'll add them to tests(7) and to an example kyua.conf file
> in devel/kyua.
>
> test_suites.FreeBSD.allow_sysctl_side_effects
> If defined, enables tests that cause potentially harmful changes to
> globally relevant sysctl variables.  Requested by Peter for
> kern_descrip_test:maxfiles__increase
>
> test_suites.FreeBSD.fibs
> Defined to a space delimited list of fib numbers.  Tests that change
> routing tables must use these fibs instead of the default.  They will
> be skipped if the variable is undefined.  I'm working on such tests
> right now.
>
> test_suites.FreeBSD.disks
> Defined to a space delimited list of disk device nodes.  Tests that
> format, mount, write to, disable, or have any other harmful effect on
> disks must use these devices.
>
> test_suites.FreeBSD.allow_devfs_side_effects.
> If defined, enables tests that can create or destroy semipermanent
> device nodes, like disk devices.  Without this variable, tests may
> still create and destroy device nodes that are normally transient,
> like /dev/tap* and /dev/pts/*.  This variable requires further
> explanation:
[...]

I think these all sound reasonable.

Can these expected side-effects be reversed in the test cleanup?

The descriptions above should clearly mention these details. For
example, for sysctl-related tests I'd expect them to restore the
original values just fine... but maybe that's not the case in the
other examples you mention?

> test_suites.FreeBSD.unit_tests
> test_suites.FreeBSD.functional_tests
> test_suites.FreeBSD.stress_tests
> These three variables would address a different concern of Peter's.
> He was concerned that his test might require too much run time,
> annoying other users of the rest of the test suite.  By using these
> variables, a sysadmin could easily disable stress tests, which tend to
> take a very long time to run.  Conversely, unit tests are normally
> very fast.  A kyua run that only ran unit tests could easily be done
> after every buildworld, for example, while one that ran functional
> tests as well could be done perhaps several times per day.  This is
> just an idea I have.  I haven't tried it in practice.
>
> What do you think?

Doing this separation by sizes is important for the reasons you
outline, but I have a different proposal that I have had in mind for a
while and only recently started to work on.

The idea is, simply, to introduce a "size" metadata variable to the
test cases and then make every test declare if they are "small",
"medium", "large", etc. With this, the user is allowed to customize,
in kyua.conf, the timeouts for every test case size -- with some
reasonable defaults for modern machines. (I've seen this implemented
in other test suites and is an approach that works pretty well.)

The reason to do it this way is to kill the "timeout" property, which
is unreliable as it is not possible to make good predictions on run
time when you are going to run the same test on multiple
architectures. With this approach, we'd then say that qemu VMs have
much longer timeouts for every class, for example.

And lastly, we'd just need a simple "filtering" feature in the kyua
cli to allow specifying which size of tests to run (or to filter by
any other metadata property, for that matter).