TET and other testing framework for FreeBSD

[Garrett Cooper]

On Tue, Jul 22, 2008 at 2:10 AM, Alexander Leidinger
<Alexander at leidinger.net> wrote:
> Quoting "Garrett Cooper" <yanefbsd at gmail.com> (from Mon, 21 Jul 2008
> 23:43:11 -0700):
>
>> Hi Joseph (and hackers),
>>      I'm contacting Joseph primarily because I saw his name listed
>> under the TET page (http://wiki.freebsd.org/TetIntegration) as a
>> contact and was wondering whether or not there was any interest to
>> integrate test suites outside of TET into FreeBSD, but I'm also
>> involving hackers@ because there might be some references that someone
>> can provide me for other @freebsd folks.
>
> AFAIK there's no TET stuff for FreeBSD available in public. So I would say
> it's more having regression tests available (making them compile and run on
> FreeBSD, having a look at the failing tests and see if it is a problem with
> the test or with FreeBSD ... see below for my experience with the LTP) at
> all, than integrating them.

Ok. Sounds like a reasonable requirement for any test infrastructure.

>>      I ask because my work with LTP (Linux Test Project :
>> http://ltp.sf.net) for Cisco has allowed me some insight into using
>> OpenPOSIX and Ballista testing frameworks, which may prove helpful in
>> the release testing cause, and could help in detecting faults earlier
>> on, thus helping expedite the release process a bit more and
>> increasing confidence and interest in FreeBSD.
>
> Do you have any URLs or whatever besides the LTP one (where those are
> integrated) to get those tests from an official point instead from cutting
> it out of the LTP?

I've contacted the PI and he said the the project is essentially dead
from his end (not enough resources to maintain, grad students `slaves'
left the project and got jobs =), etc). Currently the best way to grab
the sources is just to peel away the snapshot from LTP as it's the
only publicly and most available copy online (I believe that there've
been a few fixes tossed into Ballista as-is).

Funny thing is that a lot of groups contributed to the project, it's
GPL based, yet nothing has changed in the source for quite some
time...

>>      I'm pretty sure I have the blessings of the Ballista project's
>> principal investigator to relicense it under the BSD / GPL license,
>> it's currently a 'dead project' and he gave his blessings to post the
>> source up on Sourceforge, but I'll double check to make sure I have
>> his AOK before doing so...
>
> I think it would be beneficial so the what this produces on FreeBSD (the
> license doesn't matter here) before we go and talk about integration (here
> the license matters).

Indeed.

>>      I'm also working on getting tst_res (1.) dually licensed from
>> the LTP folks with a BSD / GPL license to allow for better inclusion
>> into FreeBSD's infrastructure, to ensure that we have a deterministic
>> means of dealing with testcases and reporting.
>> Just wondering what, if any, interest would be in adopting and
>> applying this work to FreeBSD for the good of the community.
>
> If this is what is used in the LTP to run some tests, I have to say it does
> not look mature. I've run the LTP a lot in FreeBSD (native linux binaries,
> http://wiki.FreeBSD.org/linux-kernel/ltp ... LTP not run and updated the
> page since a long time) to test the linux compatibility layer, and the are
> several places where tests fail and no output is produced or even the
> summary said the test passed. I even opened bug reports at the LTP page on
> SF, but it seems nobody was interested in those reports.

tst_res consists of the functions:

tst_res
tst_resm
tst_brk
tst_brkm
tst_brkloop
tst_brkloopm
tst_exit

which (thanks to some feedback) has finally made it up onto its own
online manpage:

http://ltp.sourceforge.net/ltpmantemp.php?file=man3/tst_res.html

This is the simple output and behavior modification API's that produce
messages like (according to the sample on the manpage):

tsttcs01    1       PASS  :  Able to create MAXUP processes
tsttcs01    2       FAIL  :  Too many processes (MAXUP+1) created
tsttcs01    3       BROK  :  tabinfo(PROCTAB, &tbs) failed; errno =
13: Permission denied
tsttcs01    4-10    BROK  :  Remaining cases broken
tsttcs01    0       WARN  :  cleanup(): kill(0, SIGALRM) failed; errno
= 3: No such process

> There are also some other ideas, like using the protocol the perl test
> suites use, to be able to use existing perl stuff to generate reports and
> overviews out of the generated logs (AFAIR this was the idea behind some
> changes to the existing regression tests a long time ago).

I have yet to really explore Test::Harness, but while I like Perl and
I'm sure that Test::Harness is as good as people laud it to be, I find
dealing with Perl to be cumbersome as they tend to break some stuff
between releases =(...

Cheers,
-Garrett