Kyua/ATF as a test framework discussion

Tue Nov 4 19:16:34 UTC 2014

On Tue, Nov 4, 2014 at 11:38 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
> (developers BCCed; please continue discussion on -testing@)
> Hi all,
>
>         A few questions came up yesterday at the BSD Vendor Summit in the Q&A session after rodrigc’s talk about Kyua/ATF related to testing on FreeBSD, as related to the testing framework (there are a handful of other questions I’ll bring up in other threads ;)..). In particular, some of the questions that stood out in my mind were:
>
>         1. What type of testing does Kyua do? Can it be used in benchmark testing, stress testing, etc?
>         2. Is there a structured way of checking for feature <X> before running my tests?
>         3. Can I run tests with custom arguments, e.g. point something at a filesystem path, etc? How do I do that?
>
>         I’ll try to answer these questions based on experience and input I’ve received from several people both internal and external to EMC/Isilon.
>
> Answer to 1.
>
> - Kyua/ATF excels as a unit testing framework and only does non-distributed testing.
> - Kyua/ATF should not be used in benchmark testing or stress testing.
>
>         There is a key component missing from kyua that allows it to run functional tests (both on FreeBSD and in general). In particular it does not have per test program setup/cleanup hooks that other frameworks do (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase setup hook. This is something that I called out several years ago when I first looked at importing ATF into the base tree (see the “Known Issues” section in https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup hooks has made some testing at EMC/Isilon painful because of how some testers have written tests, in particular there are testcases that were added to functionally test out mountd, mount_nfs, syslogd, etc that I had to disable because of the unnecessary complexity involved in trying to manage state in setting up/tearing down daemons, running syslogd, etc, and because of the external state involved the testcases would work sometimes if run end-to-end, not other times if executed one-by-one, and not other times if run end-to-end.
>
>         Adding these hooks isn’t impossible and should be the next step in improving the test Kyua/ATF infrastructure (I ran out of time before trying to implement this, and I’m not a C++ whiz so I got lost trying to figure out how to make everything work).

I partially disagree with this answer.  While it's true that per-test
program setup and teardown steps would be convenient, they are by no
means required.  Spectra Logic has many functional and stress tests
that work just fine.  Code duplication due to lack of per-test program
setup and teardown steps is minimal.  For example, see
sys/netinet/fibs_test on a CURRENT system.  The ZFS test suite (
base/projects/zfsd/head/tests/sys/cddl/zfs ) was adapted from STF,
which uses per-test program setup and cleanups.  But it turned out
that it didn't really need them, and was using them wrongly as often
as not.  For example, many test cases could only be run in a certain
order because of interactions in their side effects.  I converted all
of the setup and teardown steps to per-testcase.

OTOH, I do agree that Kyua should not be used for benchmarking.  Its
reports are much too coarse.  For one thing, a Kyua test can only pass
or fail (or skip, expected fail, etc), whereas a benchmark normally
returns one or more real numbers.  For another, in order to compare
benchmarks from different systems, it is necessary to collect a great
deal of information about the configuration of those systems.  How
much RAM?  What model of CPU?  What were the network card's sysctl
settings?  Kyua doesn't try to do that, and I don't think that it ever
should.

>
> Answer to 2.
>
>         There isn’t currently a structured API for asking questions like “is this driver loaded?", “is this sysctl set?”, “am I running on UFS/ZFS?”, etc. In general test cases from tools/regression, etc have invented homegrown logic for handling this, so this isn’t a new issue.
>
>         I’ve done some initial prototyping of a C/shell library for testing for feature requirements on FreeBSD, but the work isn’t complete/in the tree yet (please see the mailing list threads below for more details):
>
> - https://lists.freebsd.org/pipermail/freebsd-testing/2014-June/000439.html
> - https://lists.freebsd.org/pipermail/freebsd-testing/2014-July/000467.html
>
>         I want to complete this sometime before/around the start of the 2015, in particular because this blocks several of the testcases at EMC/Isilon from being upstreamed in a good, working state.
>
> Answer to 3.
>
>         Yes and no(-ish).
>
> For ATF tests:
>
>         You can pass through custom arguments on the command line like so:
>
>         % kyua test -v mykey=myvalue
>
>         This information is that passed through the atf APIs so you can query the value with either atf-config or atf_tc_config (unfortunately the mechanism I’m used to differs enough from what is currently in place, so I’ll need to go look up what the current mechanism/structure is for this, but basically it involves specifying variables kyua.conf).

Also, you can put those settings in /usr/local/etc/kyua/kyua.conf
instead of on the kyua command line.

>
> For non-ATF tests:
>
>         You have to resort to using environment variables today, like I did with pjdfstest: https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest/tests/conf?revision=274016&view=markup#l5 , as I described in the README file: https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?revision=274016&view=markup . I’d _really_ like to avoid doing this though because then we’ll have 40 some different ways to do things around the tree instead of a single, supported way to do things.
>
>         There was an enhancement I opened on GitHub to add “configuration integration” for the other test types (Plain, TAP) [via environment variables or command querying], but that wasn’t completed: https://github.com/jmmv/kyua/issues/84 . I’ll reopen the discussion.
>
>         I’ll try to add/migrate relevant links for things noted in this email on the wiki.
>
> Thank you!

-Alan