Python unittest backend for Kyua

[Alan Somers]

On Wed, Nov 27, 2013 at 6:56 AM, Julio Merino <julio at meroh.net> wrote:
> On Tue, Nov 26, 2013 at 11:47 PM, Alan Somers <asomers at freebsd.org> wrote:
>> I wrote a Kyua backend for Python programs that use the unittest
>> module from the standard library.  It's incomplete (no man page, no
>> tests, no test case isolation, and no tested version of python other
>> than 2.7), but I'm posting it to discuss the approach.  Do you think
>> it's a good start?
>>
>> Rather than use unittest's CLI, I wrote the backend to dynamically
>> load the file under test and interrogate the test using unittest as a
>> library.  This allowed more intimate access.  For example, the
>> unittest CLI doesn't have a "list" command, but the backend can still
>> list tests.
>>
>> Does it look good so far?
>
> I like the idea a lot (and I had not considered it earlier).  The code
> is a good start, but see below.
>
> One concern I have is regarding how you will implement isolation
> features.  The more I think about them, the more I think that they are
> not specific to the ATF interface at all: i.e. all tests should be
> executed with the same level of isolation.
>
> Now, in your case, it seems you'd have to reimplement these features
> in Python -- which means that there will be two different
> implementations for the same thing and this can easily lead to
> inconsistencies.  (atf had 3 versions of this in the very beginning,
> one for each supported language, and it wasn't pretty!)  But having a
> pure Python binary has its benefits, if only for simplicity.

The other benefit of pure Python is that Python doesn't become a
build-time dependency of kyua-testers.  As long as the tester is pure
python, then Python is only a runtime dependency, and it can be
ignored if you don't have any Python test programs.

>
> The alternatives would be to either 1) hook the isolation C code into
> a Python module and call it from your tester, or 2) write the tester
> in C using all existing code (including CLI parsing for a consistent
> interface, isolation features, etc.) and just call into the Python
> interpreter to list tests or run one of them.

I've never programmed around the Python/C interface.  I'll take a look
and see how hard it would be.  Is there any list of which isolation
features a tester needs?  The ones I know of are:

* Separate process for each test case
* Catch segfaults and other test deaths
* setuid
* setgid
* timeout
* chdir
* sanitize ENV variables
* core size
* reset signal handlers
* umask

-Alan

>
> What do you think?
>
> --
> Julio Merino / @jmmv
>
> --
> You received this message because you are subscribed to the Google Groups "kyua-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kyua-discuss+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.