NVMe unit tests

Craig Rodrigues rodrigc at crodrigues.org
Thu Jul 30 12:20:44 UTC 2015

On Wed, Jul 29, 2015 at 3:02 PM, Alan Somers <asomers at freebsd.org> wrote:

> On Wed, Jul 29, 2015 at 11:59 AM, Jim Harris <jimharris at freebsd.org>
> wrote:
> Kyua doesn't natively understand CUnit.  So you have two options: run the
> tests with the plain test adapter, or convert them all to ATF-C tests.  The
> main disadvantage to the plain test adapter is that Kyua won't be aware of
> the detailed test results.  For example, if you have 500 CUnit test cases,
> Kyua will only report whether the whole suite passed or failed rather than
> whether each test case passed or failed.

In the May/June 2015 issue of FreeBSD Journal ( http://freebsdjournal.com
), there is
a good interview with Julio Merino (jmmv).  In the interview, Julio says:

"If you happen to be an atf-sh user, I'd like ot suggest you have a look at
shtk's unittest module, which is my latest attempt at implementing a more
modern testing library for the shell that follows the common xUnit idioms.
And if you use atf-c++, consider googletest as an alternative, a much more
mature C++ testing library".

On a similar note, I have used CUnit, and find it to be a perfectly fine
for writing unit tests in C.  CUnit and googletest can both generate xUnit
XML test result
output which can be used as input to continuous integration systems such as

Based on the comments from Julio, I have a very hard time encouraging people
to write new tests using ATF.  I have a harder time encouraging people to
existing tests from other test libraries to ATF, if they have stuff already
working with things like
googletest or CUnit.

At my work, I originally tried to steer people to write new tests in ATF
and migrate tests from CUnit to ATF, but then
gave up, because of pushback from developers.  The combination we have that
seems to work for us is:  CUnit for unit tests in C, pyunit tests in Python
for tests which can run in a scripting language, and all tests results
reported in Jenkins.  This combination may not work for everyone, but it is
what seems to be OK for us.

If kyua could be extended to natively run CUnit and googletest, that would
very useful.  That would allow us to run more types of test libraries and
have the test results reported natively in kyua.  I don't have the time to
work on extending kyua in this way.
I'm not sure if Julio does either.  In that interview, he states that he is
quite busy.

The alternative is to run CUnit or googletests directly (even if they are
in the FreeBSD src tree), and report the results in the Jenkins test report
viewer.  That option may not be attractive, but it would work.


More information about the freebsd-testing mailing list