Kyua/ATF as a test framework discussion
yaneurabeya at gmail.com
Tue Nov 4 18:38:07 UTC 2014
(developers BCCed; please continue discussion on -testing@)
A few questions came up yesterday at the BSD Vendor Summit in the Q&A session after rodrigc’s talk about Kyua/ATF related to testing on FreeBSD, as related to the testing framework (there are a handful of other questions I’ll bring up in other threads ;)..). In particular, some of the questions that stood out in my mind were:
1. What type of testing does Kyua do? Can it be used in benchmark testing, stress testing, etc?
2. Is there a structured way of checking for feature <X> before running my tests?
3. Can I run tests with custom arguments, e.g. point something at a filesystem path, etc? How do I do that?
I’ll try to answer these questions based on experience and input I’ve received from several people both internal and external to EMC/Isilon.
Answer to 1.
- Kyua/ATF excels as a unit testing framework and only does non-distributed testing.
- Kyua/ATF should not be used in benchmark testing or stress testing.
There is a key component missing from kyua that allows it to run functional tests (both on FreeBSD and in general). In particular it does not have per test program setup/cleanup hooks that other frameworks do (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase setup hook. This is something that I called out several years ago when I first looked at importing ATF into the base tree (see the “Known Issues” section in https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup hooks has made some testing at EMC/Isilon painful because of how some testers have written tests, in particular there are testcases that were added to functionally test out mountd, mount_nfs, syslogd, etc that I had to disable because of the unnecessary complexity involved in trying to manage state in setting up/tearing down daemons, running syslogd, etc, and because of the external state involved the testcases would work sometimes if run end-to-end, not other times if executed one-by-one, and not other times if run end-to-end.
Adding these hooks isn’t impossible and should be the next step in improving the test Kyua/ATF infrastructure (I ran out of time before trying to implement this, and I’m not a C++ whiz so I got lost trying to figure out how to make everything work).
Answer to 2.
There isn’t currently a structured API for asking questions like “is this driver loaded?", “is this sysctl set?”, “am I running on UFS/ZFS?”, etc. In general test cases from tools/regression, etc have invented homegrown logic for handling this, so this isn’t a new issue.
I’ve done some initial prototyping of a C/shell library for testing for feature requirements on FreeBSD, but the work isn’t complete/in the tree yet (please see the mailing list threads below for more details):
I want to complete this sometime before/around the start of the 2015, in particular because this blocks several of the testcases at EMC/Isilon from being upstreamed in a good, working state.
Answer to 3.
Yes and no(-ish).
For ATF tests:
You can pass through custom arguments on the command line like so:
% kyua test -v mykey=myvalue
This information is that passed through the atf APIs so you can query the value with either atf-config or atf_tc_config (unfortunately the mechanism I’m used to differs enough from what is currently in place, so I’ll need to go look up what the current mechanism/structure is for this, but basically it involves specifying variables kyua.conf).
For non-ATF tests:
You have to resort to using environment variables today, like I did with pjdfstest: https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest/tests/conf?revision=274016&view=markup#l5 , as I described in the README file: https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?revision=274016&view=markup . I’d _really_ like to avoid doing this though because then we’ll have 40 some different ways to do things around the tree instead of a single, supported way to do things.
There was an enhancement I opened on GitHub to add “configuration integration” for the other test types (Plain, TAP) [via environment variables or command querying], but that wasn’t completed: https://github.com/jmmv/kyua/issues/84 . I’ll reopen the discussion.
I’ll try to add/migrate relevant links for things noted in this email on the wiki.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
More information about the freebsd-testing