Python unittest backend for Kyua

[Julio Merino]

On Tue, Nov 26, 2013 at 11:47 PM, Alan Somers <asomers at freebsd.org> wrote:
> I wrote a Kyua backend for Python programs that use the unittest
> module from the standard library.  It's incomplete (no man page, no
> tests, no test case isolation, and no tested version of python other
> than 2.7), but I'm posting it to discuss the approach.  Do you think
> it's a good start?
>
> Rather than use unittest's CLI, I wrote the backend to dynamically
> load the file under test and interrogate the test using unittest as a
> library.  This allowed more intimate access.  For example, the
> unittest CLI doesn't have a "list" command, but the backend can still
> list tests.
>
> Does it look good so far?

I like the idea a lot (and I had not considered it earlier).  The code
is a good start, but see below.

One concern I have is regarding how you will implement isolation
features.  The more I think about them, the more I think that they are
not specific to the ATF interface at all: i.e. all tests should be
executed with the same level of isolation.

Now, in your case, it seems you'd have to reimplement these features
in Python -- which means that there will be two different
implementations for the same thing and this can easily lead to
inconsistencies.  (atf had 3 versions of this in the very beginning,
one for each supported language, and it wasn't pretty!)  But having a
pure Python binary has its benefits, if only for simplicity.

The alternatives would be to either 1) hook the isolation C code into
a Python module and call it from your tester, or 2) write the tester
in C using all existing code (including CLI parsing for a consistent
interface, isolation features, etc.) and just call into the Python
interpreter to list tests or run one of them.

What do you think?

-- 
Julio Merino / @jmmv