Kyua/ATF as a test framework discussion

Craig Rodrigues rodrigc at
Thu Nov 6 02:02:20 UTC 2014

On Wed, Nov 5, 2014 at 3:45 PM, Julio Merino <jmmv at> wrote:

> See my other mail in this thread. Basically, I believe the current ATF
> interface is not worth fixing to support these cases and we should
> look at other existing, unit-testing libraries to see if they fit our
> needs. Kyua should be the glue to bring them together. In fact, you
> should be able to use all the libraries you mentioned with Kyua
> already (or at least that's the goal); and, if not, that's a bug that
> we should fix.

I'm confused by your e-mail.  Are you not interested in fixing/enhancing
and want to move to "something else"?  Is "something else" an incremental
on ATF, or something completely different and incompatible?

After Garrett's recent work to import NetBSD tests into FreeBSD, we now have
over 3000 tests in FreeBSD, many of which are ATF tests.  I hope to see
writing more tests using ATF, but they will not be inclined to do so if it
is perceived as a dead end.
Also, keep in mind, we have a few companies building BSD-based products who
started using ATF and have some interest in it.

kyua as it exists right now is mostly OK, and the "kyua + Jenkins"
integration has been
working out quite well, at least for
At the recent FreeBSD Devsummit, I presented what I did:
There was a lot of interest in that work.  There are a few companies that
want to look at that
and use it as a model for testing their own products which are based on BSD.

Integrating non-ATF test frameworks in kyua is not seamless right now.
If I treat each non-ATF test framework as a "plain_test_program" in kyua,
that works mostly OK.
However, with "plain_test_program", I don't get the ability to have
multiple distinct test cases in a single program,
like I do with ATF, which are visible via "kyua list, kyua debug", etc.
Improving things in this area is doable,
for example taking Alan's Python unittest backend to kyua (
and polishing it up.

Also, with "plain_test_program", if I want custom setup/teardown, I need to
implement that setup/teardown inside the test
program.  I don't always want that, and being able to maybe specify the
custom setup/teardown routines in the Kyuafile would
actually be useful.

ATF and Kyua are not perfect, but they are actually very, very good tools,
and are taking BSD in a very
good direction with respect to testing.  Fixing some of the issues in ATF,
and making Kyua integrate better
with other frameworks will only make it better.  I hope you are still
interested in doing some of these things.


More information about the freebsd-testing mailing list