Adding Python Wrapper for ATF

[Alan Somers]

On Fri, May 9, 2014 at 3:26 PM, Walker, Myron <myron.walker at isilon.com> wrote:
> I have created a python wrapper atfpy-sh that will act as an atf program and wrap a python module that contains unittest test cases.  I have included a zip of the wrapper script and the man page for it.  I believe it will be very efficient to create python unittest test cases and debug them in an full IDE environment like eclipse and PyDev on any linux or unix desktop system against shared libraries that are common between operating systems.  Then just add them to the FreeBSD source tree as ATF wrapped tests once they are fully operational tests.
>
> See the attached zip for the atfpy-sh script, manpage, and an example unittest python module.  I was putting it under contrib/atf/atfpy-sh and adding a Makefile under usr.bin/atf/atfpy-sh to copy atfpy-sh to /usr/bin.  Then just stick the test file in a folder under /user/tests, modify the Kyuafile for the folder and it should work.
>
> I think this would make a great addition to ATF.  I would really like to be able to use it.
>
> Thanks,
> Myron

Firstly, it looks like the list server stripped out your attachment.
Could you try posting it online somewhere?

Secondly, this sounds like it would affect more then just FreeBSD.  I
suggest moving the thread to kyua-discuss at googlegroups.com.  See also
a previous discussion about a python API at
https://groups.google.com/forum/#!topic/kyua-discuss/ji8fWB9chKg .

I never finished that API.  Partly, it's because it was really just a
toy for me; I have no serious applications for Python and Kyua.  But
it also reflected a philosophical disagreement between myself and
Julio, which I never explicitly raised.  Now seems like a good time:

What is the proper amount of test case isolation for Kyua tests
written in Python?  Julio would say that the Python API should offer
exactly the same isolation features as are found in the C and sh APIs.
 That would be most convenient if one were using Python as an
alternative to sh, to write system-level functional tests.  But a
career Pythoner would probably disagree.  If your intention is to use
Kyua to execute the unit tests for some existing Python program, then
you would probably want exactly the same isolation features as are
offered by Python's standard unittest module.  For example, if you use
a Python program from ports, then you might want to install its unit
tests into /usr/tests and run them along with FreeBSD's own tests.  In
such a case, Kyua's extra isolation features would be superfluous
because the tests were written without them in mind.  The isolation
features might even cause some test cases to fail, if they use the
class and module level fixtures introduced in Python 2.7.

Even if your tests would work fine with all of the isolation features,
then would run much slower without.  Kyua creates a separate process
for each test case and each test case teardown.  That works fine in sh
and C, but incurs considerable overhead in languages like Python,
Ruby, and Java.

If we can agree on an answer to this question, then I think we'll be
able to add APIs for many more languages, and greatly increase Kyua's
usefulness.  My personal preference is for APIs that predate Kyua,
like Python's unittest, Ruby's unit/test, or C++'s googletest to
execute the tests as similarly as possible whether or not Kyua is
involved.  That would mean no extra isolation.  People who do want the
isolation, for example to test external binaries using Python as glue
logic, can always use the ATF sh API and call a python program.  But
there's certainly room for disagreement.  I could almost go the other
way for googletest, since its startup time is so low.

-Alan

>
> _______________________________________________
> freebsd-testing at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-testing
> To unsubscribe, send any mail to "freebsd-testing-unsubscribe at freebsd.org"