Incorporating the Capsicum test suite

[Craig Rodrigues]

On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste <emaste at freebsd.org> wrote:

>
> I have a couple of questions. First, do we have an existing pattern
> for connecting googletest tests to Kyua?
>

No we do not.  Googletest looks like it is basically trying to do
the same thing as ATF or CUnit.

Googletest has a way to generate JUnit XML test output:
https://code.google.com/p/googletest/wiki/AdvancedGuide#Generating_an_XML_Report

The easiest thing to do would be to configure the Capsicum tests to
generate this XML test output.  This can be imported natively into
Jenkins, which can then parse the test report.
This would require no changes to the Capsicum tests.

If you want to integrate googletest with kyua, then things get tricky.
On the googletest page: https://code.google.com/p/googletest/
they list a third party plugin which can take googletest and generate TAP
output.
I don't know how well that works.  It would be nice if that functionality
was
integrated in googletest natively.

Another option would be to modify kyua, so that it can parse the output of
googletest
natively.  This would be quite nice.  I haven't found extending kyua to be
very
easy....it requires good knowledge of C++ and Lua.  Also, whoever
contributes
code to kyua must sign a Contributor License Agreement (CLA) with Google.
That's something to keep in mind.

Another option would be to rewrite the Capsicum tests in ATF.
I don't know how many tests there are, but this might be doable.
ATF and Googletest look similar.

If you wanted to import the Capsicum tests into FreeBSD as they are right
now, then it looks
like you would need to import googletest into FreeBSD as well.  googletest
is BSD licensed,
so that is not a problem, but we already have ATF in the tree, so I don't
know how
useful it would be to have multiple test frameworks in the base system.

--
Craig