TET and other testing framework for FreeBSD

[Alexander Leidinger]

Quoting "Garrett Cooper" <yanefbsd at gmail.com> (from Mon, 21 Jul 2008  
23:43:11 -0700):

> Hi Joseph (and hackers),
>       I'm contacting Joseph primarily because I saw his name listed
> under the TET page (http://wiki.freebsd.org/TetIntegration) as a
> contact and was wondering whether or not there was any interest to
> integrate test suites outside of TET into FreeBSD, but I'm also
> involving hackers@ because there might be some references that someone
> can provide me for other @freebsd folks.

AFAIK there's no TET stuff for FreeBSD available in public. So I would  
say it's more having regression tests available (making them compile  
and run on FreeBSD, having a look at the failing tests and see if it  
is a problem with the test or with FreeBSD ... see below for my  
experience with the LTP) at all, than integrating them.

>       I ask because my work with LTP (Linux Test Project :
> http://ltp.sf.net) for Cisco has allowed me some insight into using
> OpenPOSIX and Ballista testing frameworks, which may prove helpful in
> the release testing cause, and could help in detecting faults earlier
> on, thus helping expedite the release process a bit more and
> increasing confidence and interest in FreeBSD.

Do you have any URLs or whatever besides the LTP one (where those are  
integrated) to get those tests from an official point instead from  
cutting it out of the LTP?

>       I'm pretty sure I have the blessings of the Ballista project's
> principal investigator to relicense it under the BSD / GPL license,
> it's currently a 'dead project' and he gave his blessings to post the
> source up on Sourceforge, but I'll double check to make sure I have
> his AOK before doing so...

I think it would be beneficial so the what this produces on FreeBSD  
(the license doesn't matter here) before we go and talk about  
integration (here the license matters).

>       I'm also working on getting tst_res (1.) dually licensed from
> the LTP folks with a BSD / GPL license to allow for better inclusion
> into FreeBSD's infrastructure, to ensure that we have a deterministic
> means of dealing with testcases and reporting.
> Just wondering what, if any, interest would be in adopting and
> applying this work to FreeBSD for the good of the community.

If this is what is used in the LTP to run some tests, I have to say it  
does not look mature. I've run the LTP a lot in FreeBSD (native linux  
binaries, http://wiki.FreeBSD.org/linux-kernel/ltp ... LTP not run and  
updated the page since a long time) to test the linux compatibility  
layer, and the are several places where tests fail and no output is  
produced or even the summary said the test passed. I even opened bug  
reports at the LTP page on SF, but it seems nobody was interested in  
those reports.

There are also some other ideas, like using the protocol the perl test  
suites use, to be able to use existing perl stuff to generate reports  
and overviews out of the generated logs (AFAIR this was the idea  
behind some changes to the existing regression tests a long time ago).

Bye,
Alexander.

-- 
I read Playboy for the same reason I read National Geographic.
To see the sights I'm never going to visit.

http://www.Leidinger.net    Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org       netchild @ FreeBSD.org  : PGP ID = 72077137