TET and other testing framework for FreeBSD

[Garrett Cooper]

On Tue, Jul 22, 2008 at 10:21 PM, Alexander Leidinger
<Alexander at leidinger.net> wrote:
> Quoting "Garrett Cooper" <yanefbsd at gmail.com> (from Tue, 22 Jul 2008
> 08:31:12 -0700):
>
>> On Tue, Jul 22, 2008 at 2:10 AM, Alexander Leidinger
>> <Alexander at leidinger.net> wrote:
>>>
>>> Quoting "Garrett Cooper" <yanefbsd at gmail.com> (from Mon, 21 Jul 2008
>>> 23:43:11 -0700):
>
>>>>     I'm also working on getting tst_res (1.) dually licensed from
>>>> the LTP folks with a BSD / GPL license to allow for better inclusion
>>>> into FreeBSD's infrastructure, to ensure that we have a deterministic
>>>> means of dealing with testcases and reporting.
>>>> Just wondering what, if any, interest would be in adopting and
>>>> applying this work to FreeBSD for the good of the community.
>>>
>>> If this is what is used in the LTP to run some tests, I have to say it
>>> does
>>> not look mature. I've run the LTP a lot in FreeBSD (native linux
>>> binaries,
>>> http://wiki.FreeBSD.org/linux-kernel/ltp ... LTP not run and updated the
>>> page since a long time) to test the linux compatibility layer, and the
>>> are
>>> several places where tests fail and no output is produced or even the
>>> summary said the test passed. I even opened bug reports at the LTP page
>>> on
>>> SF, but it seems nobody was interested in those reports.
>>
>> tst_res consists of the functions:
>>
>> tst_res
>> tst_resm
>> tst_brk
>> tst_brkm
>> tst_brkloop
>> tst_brkloopm
>> tst_exit
>>
>> which (thanks to some feedback) has finally made it up onto its own
>> online manpage:
>>
>> http://ltp.sourceforge.net/ltpmantemp.php?file=man3/tst_res.html
>>
>> This is the simple output and behavior modification API's that produce
>> messages like (according to the sample on the manpage):
>>
>> tsttcs01    1       PASS  :  Able to create MAXUP processes
>> tsttcs01    2       FAIL  :  Too many processes (MAXUP+1) created
>> tsttcs01    3       BROK  :  tabinfo(PROCTAB, &tbs) failed; errno =
>> 13: Permission denied
>> tsttcs01    4-10    BROK  :  Remaining cases broken
>> tsttcs01    0       WARN  :  cleanup(): kill(0, SIGALRM) failed; errno
>> = 3: No such process
>
> And this is where I've seen problems while testing the linux compatibility
> layer in FreeBSD. Some tests said PASS when in reality they didn't pass but
> died. I don't know if this is fixed now, it's been a while when I reported
> this.

Yeah, and I've seen tests hang on LTP, but that's because people don't
monitor test execution but just claim that since it works, it's fine.

I try to keep abreast of these things because test quality's
important; otherwise your results are worth no more than a $2 bill!

>>> There are also some other ideas, like using the protocol the perl test
>>> suites use, to be able to use existing perl stuff to generate reports and
>>> overviews out of the generated logs (AFAIR this was the idea behind some
>>> changes to the existing regression tests a long time ago).
>>
>> I have yet to really explore Test::Harness, but while I like Perl and
>> I'm sure that Test::Harness is as good as people laud it to be, I find
>> dealing with Perl to be cumbersome as they tend to break some stuff
>> between releases =(...
>
> I don't know if I was clear, the tests themself are written in C (or
> whatever) and not in perl, the goal as I remember it was to be able to use
> the already existing perl tools where we don't care about the development,
> as someone else will fix broken things because it is used for perl itself.
>
> Bye,
> Alexander.

Hmmm... I've also been forced into working with Python nose lately,
and while I might not be a Python officiando, I do find the framework
to be expeditious and simple to use.

Just another thought, because Python seems to be a better base framework...

-Garrett