Python unittest backend for Kyua

[Julio Merino]

> On Feb 15, 2015, at 00:16, Alan Somers <asomers at freebsd.org> wrote:
> 
> Not much further than when I made that post, and no.  In fact, I
> hardly use Python at all.  Professionally, my company is all about
> Ruby, and I would be very interested in a Ruby test/unit tester for
> Kyua.  However, Ruby's test/unit's implementation is much messier than
> Python's unittest, so I thought that writing a unittest tester would
> be good practice for writing a test/unit tester.  Indeed it was.  I
> found unittest very easy to interface to; it hardly took me any time
> to write that tester.  But in the end I stopped working on it because
> jmmv and I disagreed about test isolation.  Jmmv's primary interest is
> in writing system-level tests.  He basically sees a unittest tester as
> an alternative to atf-sh.  But my primary interest is in tying
> together separate components' test suites and get a consistent view of
> all results. So naturally jmmv wants the same level of isolation as
> atf-sh provides, but I want the same level of isolation as unittest
> provides.  In fact, atf-sh style isolation is bad for my use case,
> because it can cause tests that were originally written for unittest
> to fail.  It also drastically increases runtime because the Python
> interpreter must be restarted between each test.

As a matter of fact, I have changed my thoughts on this.  I would like Kyua to better support "unittest"-style test programs because that's what pretty much all testing libraries implement and what other people know... and well, this model is just much faster and equally useful for the vast majority of the cases.

In fact, ATF's interface is overkill... and super-complicated to implement in the run-time engine.

Note that the "unittest" attempt I did in shtk 1.6 does not follow any of the ATF conventions and I think it's more manageable than atf: https://groups.google.com/d/msg/kyua-discuss/yhcDuchVtrs/D6C_LU6FXOUJ

> I've never used nose or pytest.  Personally, I'm a little leery of
> test discovery, because it's contrary to how Kyua normally works.
> Normally, you have to explicitly specify each test file in the
> Kyuafile (or the makefile that builds the Kyuafile).  Having some
> tests be automatically discovered and others specified in Kyuafiles
> might be confusing.  I'm leaning against automatic discovery.  But I
> could be convinced otherwise.  In the case of Ruby, all of our tests
> are executed through Rake.  So my preferred method would be to add a
> Rake task that would generate the appropriate Kyuafile.

I think auto-discovery within the test program is great although, unfortunately, this is something that ATF does not provide.  (It'd be baked in but I have not thought about how hard that would be.)

I do not like the idea of automatic test program discovery, but Kyuafiles offer you enough functionality on their own (I think) to do this yourself.  See the Kyuafile.top file shipped in Kyua.

Cheers