Python unittest backend for Kyua
asomers at freebsd.org
Wed Nov 27 23:47:27 UTC 2013
On Wed, Nov 27, 2013 at 6:56 AM, Julio Merino <julio at meroh.net> wrote:
> On Tue, Nov 26, 2013 at 11:47 PM, Alan Somers <asomers at freebsd.org> wrote:
>> I wrote a Kyua backend for Python programs that use the unittest
>> module from the standard library. It's incomplete (no man page, no
>> tests, no test case isolation, and no tested version of python other
>> than 2.7), but I'm posting it to discuss the approach. Do you think
>> it's a good start?
>> Rather than use unittest's CLI, I wrote the backend to dynamically
>> load the file under test and interrogate the test using unittest as a
>> library. This allowed more intimate access. For example, the
>> unittest CLI doesn't have a "list" command, but the backend can still
>> list tests.
>> Does it look good so far?
> I like the idea a lot (and I had not considered it earlier). The code
> is a good start, but see below.
> One concern I have is regarding how you will implement isolation
> features. The more I think about them, the more I think that they are
> not specific to the ATF interface at all: i.e. all tests should be
> executed with the same level of isolation.
> Now, in your case, it seems you'd have to reimplement these features
> in Python -- which means that there will be two different
> implementations for the same thing and this can easily lead to
> inconsistencies. (atf had 3 versions of this in the very beginning,
> one for each supported language, and it wasn't pretty!) But having a
> pure Python binary has its benefits, if only for simplicity.
The other benefit of pure Python is that Python doesn't become a
build-time dependency of kyua-testers. As long as the tester is pure
python, then Python is only a runtime dependency, and it can be
ignored if you don't have any Python test programs.
> The alternatives would be to either 1) hook the isolation C code into
> a Python module and call it from your tester, or 2) write the tester
> in C using all existing code (including CLI parsing for a consistent
> interface, isolation features, etc.) and just call into the Python
> interpreter to list tests or run one of them.
I've never programmed around the Python/C interface. I'll take a look
and see how hard it would be. Is there any list of which isolation
features a tester needs? The ones I know of are:
* Separate process for each test case
* Catch segfaults and other test deaths
* sanitize ENV variables
* core size
* reset signal handlers
> What do you think?
> Julio Merino / @jmmv
> You received this message because you are subscribed to the Google Groups "kyua-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kyua-discuss+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
More information about the freebsd-testing