Python unittest backend for Kyua

[Alan Somers]

On Sat, Feb 14, 2015 at 2:57 PM, Craig Rodrigues <rodrigc at freebsd.org> wrote:
> On Tue, Nov 26, 2013 at 09:47:29PM -0700, Alan Somers wrote:
>> I wrote a Kyua backend for Python programs that use the unittest
>> module from the standard library.  It's incomplete (no man page, no
>> tests, no test case isolation, and no tested version of python other
>> than 2.7), but I'm posting it to discuss the approach.  Do you think
>> it's a good start?
>>
>> Rather than use unittest's CLI, I wrote the backend to dynamically
>> load the file under test and interrogate the test using unittest as a
>> library.  This allowed more intimate access.  For example, the
>> unittest CLI doesn't have a "list" command, but the backend can still
>> list tests.
>>
>> Does it look good so far?
>>
>> -Alan
>
> Alan,
>
> I saw this post of yours from a while back.
> I was just wondering, how far did you get with it?  Do you use
> it in your company?


Not much further than when I made that post, and no.  In fact, I
hardly use Python at all.  Professionally, my company is all about
Ruby, and I would be very interested in a Ruby test/unit tester for
Kyua.  However, Ruby's test/unit's implementation is much messier than
Python's unittest, so I thought that writing a unittest tester would
be good practice for writing a test/unit tester.  Indeed it was.  I
found unittest very easy to interface to; it hardly took me any time
to write that tester.  But in the end I stopped working on it because
jmmv and I disagreed about test isolation.  Jmmv's primary interest is
in writing system-level tests.  He basically sees a unittest tester as
an alternative to atf-sh.  But my primary interest is in tying
together separate components' test suites and get a consistent view of
all results. So naturally jmmv wants the same level of isolation as
atf-sh provides, but I want the same level of isolation as unittest
provides.  In fact, atf-sh style isolation is bad for my use case,
because it can cause tests that were originally written for unittest
to fail.  It also drastically increases runtime because the Python
interpreter must be restarted between each test.


>
> I looked at three Python testing frameworks:
>      (1) unittest https://docs.python.org/2/library/unittest.html
>      (2) nose  https://nose.readthedocs.org
>      (3) pytest http://pytest.org
>
> I found that pytest was the easiest to set up and did what I needed:
>     -> test discovery
>     -> generate XML test result output compatible with Jenkins


The way I wrote my bindings, this is a non-issue.  With my tester,
kyua itself is still responsible for producing the reports.


>     -> tests do not need to inherit from any base classes,
>        and just requires a file with "test_*" methods
>
> The test discovery makes it somewhat "kyua-like".
>
>
> Do you have any opions/experience with this?

I've never used nose or pytest.  Personally, I'm a little leery of
test discovery, because it's contrary to how Kyua normally works.
Normally, you have to explicitly specify each test file in the
Kyuafile (or the makefile that builds the Kyuafile).  Having some
tests be automatically discovered and others specified in Kyuafiles
might be confusing.  I'm leaning against automatic discovery.  But I
could be convinced otherwise.  In the case of Ruby, all of our tests
are executed through Rake.  So my preferred method would be to add a
Rake task that would generate the appropriate Kyuafile.

>
> My experience is that running the FreeBSD tests in kyua work great.
> However, for a full product, I've had a really hard time getting
> QA people to be able to get comfortable writing new ATF tests and Kyuafiles.
> I've had less problems withe the Python frameworks.


My main advice is to think about whether you really want to write
tests using an existing framework, or whether by the time you finish
Kyua-ifying it won't really be the same framework anymore.  I think
that there's space for both.  What my experience taught me is that one
interface can't satisfy both needs.

-Alan


>
> --
> Craig