Incorporating the Capsicum test suite

[Ed Maste]

On 3 June 2015 at 11:30, Craig Rodrigues <rodrigc at freebsd.org> wrote:
> On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste <emaste at freebsd.org> wrote:
>>
>> I have a couple of questions. First, do we have an existing pattern
>> for connecting googletest tests to Kyua?
>
> No we do not.  Googletest looks like it is basically trying to do
> the same thing as ATF or CUnit.

Yes, the intent is the same. My desire to integrate it with Kyua is so
that we can add this test coverage without needing the user to run
additional commands, parse different output formats, etc. We could
integrate the existing tests as-is into Jenkins fairly easily, but I
also want us to move to a model where developers consistently run the
test suite prior to committing a change.

> Another option would be to modify kyua, so that it can parse the output of
> googletest
> natively.  This would be quite nice.  I haven't found extending kyua to be
> very
> easy....it requires good knowledge of C++ and Lua.  Also, whoever
> contributes
> code to kyua must sign a Contributor License Agreement (CLA) with Google.
> That's something to keep in mind.

It seems to me that this is the best way to go, with connecting the
existing tests as-is into Jenkins as my second choice. FreeBSD
developers who wish to make changes to Kyua and the Google CLA is an
issue we could fork it and make changes in our own version.  It's not
ideal, but in practice if we don't expect large, conflicting changes
from upstream it should be a workable approach.

> Another option would be to rewrite the Capsicum tests in ATF.
> I don't know how many tests there are, but this might be doable.
> ATF and Googletest look similar.

I think this is feasible, and may make sense for the Capsicum tests.
However, the idea extends beyond the Capsicum tests.  Clang/LLVM (and
LLDB) have a large test suite that I'd like incorporate into our
process, and other 3rd party software does as well.  It's not feasible
to try to convert all of these to ATF.

> If you wanted to import the Capsicum tests into FreeBSD as they are right
> now, then it looks
> like you would need to import googletest into FreeBSD as well.  googletest
> is BSD licensed,
> so that is not a problem, but we already have ATF in the tree, so I don't
> know how
> useful it would be to have multiple test frameworks in the base system.

The parts of googletest actually required for the capsicum tests are
fairly small and self-contained (even though the entire googletest
package is included in the capsicum-test repo).  Assuming
capsicum-test gets stripped it down to just what's required I might
just consider the embedded googletest part of capsicum-test, rather
than importing it as a standalone project and having capsicum-test use
it.