FreeBSD Testing Facility
Mehmet Erol Sanliturk
m.e.sanliturk at gmail.com
Sun Feb 24 20:02:06 UTC 2013
On Sun, Feb 24, 2013 at 9:25 AM, Giorgos Keramidas <keramida at freebsd.org>wrote:
> On 2013-02-21 07:04, Matthew Jacob <mjacob at freebsd.org> wrote:
> > On 2/21/2013 5:04 AM, Mehmet Erol Sanliturk wrote:
> > >Dear All ,
> > >
> > >During development of FreeBSD , testing is very vital .
> > >...
> >
> > This in general is a good suggestion. Most companies do such
> > automated testing as a matter of course.
> >
> > Note however that this is a volunteer effort. Were you volunteering
> > to set up such an automated, possibly testzilla driven, facility? It
> > would certainly help the quality, although as others have noted
> > snapshots are often likely to be broken.
>
> To the OP:
>
> Like Matthew has said, this is a volunteer effort. So if you have
> experience with setting up testing automation, and you are willing to
> help us set up something like this, please go ahead :-)
>
> I've worked in places where the following types of tests are used:
>
> - Presubmit tests that check specific parts of functionality.
>
> - Commit-related tests that run asynchronously in the background,
> and report back later (e.g. through email).
>
> - Test systems that cache previous results and report a simple 'green'
> vs. 'red' status for _every_ single commit.
>
> - System tests that check for particular functionality, health
> criteria, etc. - some times fully automated, some other times
> requiring a token amount of manual support.
>
> So here are two important questions, regarding the tests you mentioned:
>
> When you speak about 'testing FreeBSD', which type of tests are you
> interested in seeing?
>
> Are you willing to help us set up something that runs the type of tests
> that you want to see?
>
>
To another message I replied as follows :
http://lists.freebsd.org/pipermail/freebsd-current/2013-February/039984.html
I want say that again to manage such a system is not possible for me very
much , additionally for lack of facilities .
My wish is that let's take this subject to be worked in detail and by a
group effort , design , implement and apply such a testing facility .
In this work , if anything I can supply , I will never keep myself back .
Without reflective responses , if we consider my first message :
http://lists.freebsd.org/pipermail/freebsd-current/2013-February/039976.html
a very rough outline is supplied .
Reason is such a suggestion is to see many failures of *BSD systems that
these can be eliminated if such a system is designed , implemented , and
applied .
In many messages by FreeBSD developers , it is said that it is not possible
to establish a testing farm by the FreeBSD project and maintain it , which
is quite correct .
Assume the following :
If any one is downloading and trying to install and run Current development
iso files , itis very likely that the main intention is to test and help to
its development .
By counting number of such testing attempts , it is possible to estimate
the number of people which can participate to a more coordinated testing
system .
My opinion is that such a system ( a distributed testers population ) is
already present , and over time , even it may be enlarged .
Actually , problem is large enough to establish a working group to attack
it .
There is a necessity to have committers to generate the necessary ftp
sites and manage them .
At present , there is a port system . The testing facility will be similar
to that system , but different programs will be employed .
The first action will be to design what will be tested and how .
Since FreeBSD is composed from components ( boot , kernel , each system
programs , etc. , )
, every component will be a testing subject .
In the svn , there are already testing parts .
By combining these , and supplying , developing missing parts will generate
a testing framework .
At the beginning , it will not be possible to generate a state of the art
testing facility .
By starting from the existing parts , over time , it will be possible to ad
tests one by one .
This will be similar to development of ports system : Over time it has been
populated one by one .
As a an applicable first step , a FreeBSD developer , may establish a wiki
page or use an existing one to develop a "Testing Requirements and Design ,
Implementation and Applications" page .
A mailing list may be established to discuss testing problems .
An ftp site may be established to apply tests as suggested in my first
message .
As an example :
Assume that a part related to video display cards is under development ,
such as KMS .
The developer(s) will have a limited number of cards available in their
computers .
A potential people population exists which they use such cards , for
example RADEON , or INTEL cards .
In the mailing lists , it may be announced that testers are needed for
specific video card branch , such as RADEON . People , fills a form to
participate in the tests .
The developer prepares a file just like a port/package .
Sends a message to the subscribers wishing to test this problem .
Possible testers , upon receiving that message , may enter a command , such
as
# test_apply name_of_testing_package
just similar to pkg_add command .
The test_apply program , downloads the testing package and applies it .
At the end , the best action is to generate a structured e-mail , the
results may be sent to the testing data base just a like bug tracker system
.
I emphasize structured messages for their automatic processing possibility .
If this is not very feasible , at least a formal message may be requested
to evaluate results
easily . Otherwise results like "Tests failed" will not be very helpful for
the developers .
A script in there processes the returned e-mails and generates a report to
developer(s) ,.
A cycle of such tests continues up to a satisfactory result .
Some parts , such as the following , are dependent very much to hardware to
be used :
- Booting process
- Network communications
- Video display units
- USB attached components
- Parts requiring BIOS cooperation ,
- Parts detecting , managing main board parts ( circuits , ports , etc. )
- Parts detecting , managing attached peripheral devices .
All of these may be subject of such distributed testing actions .
After testing many of the components and ensuing that they are working ,
there comes to
testing of their combinations , for example :
- File systems in local disks,
- NFS like systems
Then , a whole test : Testing complete install iso files . Here , there
will not be a booting or hardware detection failure , but failures related
to cooperative working of systems under user applications .
These failures may occur between mismatched components during their
interaction .
Therefore , such a testing facility requires cooperation of many people to
function properly .
A group of leaders may organize the efforts .
Thank you very much .
More information about the freebsd-current
mailing list