Request for individuals interested in reviewing test / python topics

[Garrett Cooper]

On Sat, Nov 15, 2008 at 11:57 PM, Alexander Leidinger
<Alexander at leidinger.net> wrote:
> Quoting Garrett Cooper <yanegomi at gmail.com> (from Sat, 15 Nov 2008 14:08:02
> -0800):
>
>> Hello Hackers and Porters,
>>        I'm currently working on a proposal to the FreeBSD foundation to
>> use Python Nose as a testing framework for writing tests. If there are
>
> Are you aware of the history of the current regression tests?
>
> If not:
>
> It started without a structure, then some work was done to move to the perl
> testing framework style (really only the output of the tests, and the naming
> conventions in the directory). This was not completed, and newer tests may
> not comply.
>
> The reason for choosing the perl style was, to be able to use the extensive
> perl tools to
>  - automatically run all the tests
>  - be able to compare different runs with the perl tools
>  - be able to generate a lot of different output formats (html/text/...)
>
> There's also a wiki page about testing, which you may want to check out:
>    http://wiki.freebsd.org/TetIntegration
>
> I don't really know python nose. I just looked at it quickly and can not see
> any big benefit compared to the perl test protocol outlined above (and the
> stuff outlined in the wiki looks even more advanced than that). Would you
> please elaborate where you see the benefits of it?
>
> Note that during release building perl is needed anyway to generate the
> index for the ports collection. I don't know if python is required currently
> during the release generation.
>
> Bye,
> Alexander.

Alexander,

Thanks for getting back with me so quickly. Let me help explain a bit...

About selecting tests --

Ok. I gave a quick look at Test::Simple, Test::Harness, etc and
although Perl's Test:: modules appear to be rather expedited for
executing entire tests, it's not helpful when executing selected
tests, unless you write wrappers -- which is part of the point where
the pain in adding infrastructure comes in. The default selector
plugin that comes with nose is smart enough to crack open the files in
a directory structure, do some regexed based searching for
^\s+def\s+test, or something along those lines, and from that you can
select individual files with testcases, individual functions in files
to execute, or classes, or instances of classes, etc. So if you're
developing a fix for a feature bugfix and were provided a single
regression test to use for finding the issue and were adding that test
to a master test file (for suite execution), but instead wanted to
execute only one testcase -- nose supports that functionality out of
the box with little change for the user with the default selector
plugin:

nosetests some_file.py:some_func

Furthermore,

About capture and printout --

Also, nose with plugins gives you the ability to print not just in
plaintext output on a console, but also HTML (I tried looking for the
example on the site that was available before, but it appears to be
missing now). The example probably consisted of approximately 30~40
lines real code, which for the most part had embedded HTML formatting
tags, etc. So, a basic output capture plugins with nose that prints
out HTML can be written with <= 25 lines.

I'm not sure whether or not Python is required for release generation,
but it will have to be listed as one of the required tools in the
proposal I'm going to write. Thankfully the required dependencies for
Python are small (just a base system with the typical components), and
if using Python v2.5+, you also have access to ctypes (a means for
testing C-API's in libraries), which is incredibly beneficial I've
discovered through my work at Cisco. Assembler instructions would
potentially need to be added for architectures other than x86. I'll
have to look into this further with Python 2.5 (we currently use
Python 2.4.5 -- the last maintenance release of Python -- in our
group).

Hope that helps explain things a bit more,

-Garrett