Kyua/ATF as a test framework discussion

Julio Merino jmmv at
Wed Nov 5 21:41:39 UTC 2014

On Tue, Nov 4, 2014 at 1:38 PM, Garrett Cooper <yaneurabeya at> wrote:
> (developers BCCed; please continue discussion on -testing@)
> Hi all,
>         A few questions came up yesterday at the BSD Vendor Summit in the Q&A session after rodrigc’s talk about Kyua/ATF related to testing on FreeBSD, as related to the testing framework (there are a handful of other questions I’ll bring up in other threads ;)..). In particular, some of the questions that stood out in my mind were:
>         1. What type of testing does Kyua do? Can it be used in benchmark testing, stress testing, etc?
>         2. Is there a structured way of checking for feature <X> before running my tests?
>         3. Can I run tests with custom arguments, e.g. point something at a filesystem path, etc? How do I do that?
>         I’ll try to answer these questions based on experience and input I’ve received from several people both internal and external to EMC/Isilon.
> Answer to 1.
> - Kyua/ATF excels as a unit testing framework and only does non-distributed testing.
> - Kyua/ATF should not be used in benchmark testing or stress testing.

This is a fair description.

>         There is a key component missing from kyua that allows it to run functional tests (both on FreeBSD and in general). In particular it does not have per test program setup/cleanup hooks that other frameworks do (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase setup hook.

Well, this is specifically ATF's fault. The more I look at ATF and
other testing libraries, the more I'm convinced that ATF's approach of
forcing every test case be in a completely separate process is
suboptimal because it prevents having fixtures altogether.

If you write your own test program without ATF, then you can easily
implement your own global setup and teardown routines. (In fact, the
alternative library I'm playing with in actually supports per-test
setup/teardown and it'd be trivial to add per-program setup/teardown
as well.)

Of course, "write your own test program" is poor advice given that
doing so from scratch, without support libraries, sucks. The problem
is that I do not believe that the ATF interface is fixable to support
these cases; we'd need to write a different library, and at that
point... is it worth doing? What else is there? You should be able to
easily plug existing, more advanced libraries like pyUnit or
googletest into Kyua -- and if there is a problem with doing such
integration, we should fix it.

> Answer to 2.
>         There isn’t currently a structured API for asking questions like “is this driver loaded?", “is this sysctl set?”, “am I running on UFS/ZFS?”, etc. In general test cases from tools/regression, etc have invented homegrown logic for handling this, so this isn’t a new issue.

Having such functionality would be great, but IMHO the majority of it
is so tied to FreeBSD that it has to live in the FreeBSD source tree.

> Answer to 3.
>         Yes and no(-ish).

A major goal when designing ATF back then was precisely to avoid
variability in test execution: all test programs should behave the
same way so that their execution can be scripted and so that the
caller (user) needn't know how to run the test program (usually poorly

Aren't configuration variables passed through the environment
sufficient?  Kyua 0.11 should do that.  (Note that you say above that
#84 wasn't completed, but I'm not sure why.)

More information about the freebsd-testing mailing list