Kyua/ATF as a test framework discussion

[Craig Rodrigues]

On Tue, Nov 4, 2014 at 10:38 AM, Garrett Cooper <yaneurabeya at gmail.com>
wrote:

>         There is a key component missing from kyua that allows it to run
> functional tests (both on FreeBSD and in general). In particular it does
> not have per test program setup/cleanup hooks that other frameworks do
> (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase
> setup hook.
>
This is something that I called out several years ago when I first looked
> at importing ATF into the base tree (see the "Known Issues" section in
> https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup hooks
> has made some testing at EMC/Isilon painful because of how some testers
> have written tests, in particular there are testcases that were added to
> functionally test out mountd, mount_nfs, syslogd, etc that I had to disable
> because of the unnecessary complexity involved in trying to manage state in
> setting up/tearing down daemons, running syslogd, etc, and because of the
> external state involved the testcases would work sometimes if run
> end-to-end, not other times if executed one-by-one, and not other times if
> run end-to-end.
>
>
I've raised this issue on kyua-discuss and Julio is not in favor of it:
https://groups.google.com/forum/#!msg/kyua-discuss/6DmYvtLN66I/MuG528I1V1AJ

I agree with you and disagree with Julio.  In some cases, having the hooks
in the test framework to do setup/cleanup
outside of the test itself is useful.
My experience is that people who write tests don't always write perfect
code, and do not always do proper
setup/teardown inside the testcase.  In FreeBSD and NetBSD, we can enforce
this and force people to fix the tests.

In large products based on FreeBSD, my experience is that this is not often
possible in real-world QA environments.

Having hooks in the test framework to do setup/teardown like in other unit
test frameworks would be useful.

--
Craig