FreeBSD Testing Facility
Mehmet Erol Sanliturk
m.e.sanliturk at gmail.com
Sun Feb 24 22:49:18 UTC 2013
On Sun, Feb 24, 2013 at 12:48 PM, Alexander Yerenkow <yerenkow at gmail.com>wrote:
> How about create -testing mail list, and start searching for any kind of
> I could lend a hand in creating auto-testing some aspects of FreeBSD (at
> leas successful booting/working net/route etc),
> someone else probably could have to say something too.
> I think resources are there, they just need to be gathered :)
> Alexander Yerenkow
Really , there are ample amount of facilities in FreeBSD already .
The steps may be the following :
- Prepare a testing requirements document :
There are existing testing programs in the svn .List them .
. The developers may use tests that are not in svn . Collect information
about them .
. Collect additional testing point suggestions .
. Design a testing facility :
This may be just an imitation of the packages/ports tree .
For each test , prepare a package .
The new package management project contains very important information to
Prepare a test_apply program just like pkg_add .
Generate an ftp site parts for .../testing/Fatal/... ,
.../testing/Dangerous/... , .../testing/Harmless/...
Generate databases for the collected result mails , with respect to
message groups .
For each test , generate an ( xml , or JSON , or just name = value pairs
( like .conf files ) ) results
e-mail structure .
During tests , this file will be created .
When such an e-mail is received into a definite address , with program ,
introduce it into a database .
Generate reports from databases to submit to developers .
Generate a mailing list about testing facility .
Prepare e-mail templates with respect to testing points ( groups ,
subjects ) .
To the subscriber , present a form to fill information which parts he or
she can test , and store these into a data base .
Associate each test package with fields of the subscribers testing
When a testing action is required ( generated by a modification in a
tested area ) , query the subscribers database , and send an e-mail to
possible testers .
Before test_apply program is executed , if it is necessary to perform some
actions , prepare an algorithm to define such steps and attach to e-mail ,
or fetch by test_apply .
The above steps may be implemented mostly by facilities present in the base
system and packages in the ports tree .
Identify required new programs , write-ups , scripts , etc. , to design ,
and implement them .
Application , Testing :
By using a harmless test , apply the above designed system and evaluate
recycle the above steps ,
Start to populate the testing system with packages one by one .
Maintain the testing tree , and when a modification is applied into a part
, if necessary , also
modify the testing package and request a testing step .
Now , there are "Call for Testing" applications in the present mailing
With the above structure , such requests will be more widely applied and
will produce structured ,
analyzable results .
If one such package is developed , the rest will start to imitate its many
steps , development of new testing packages will be easy .
Some of the test packages may also be installed during a complete FreeBSD
installation and may be used to test the installed system , and the results
may be collected to analyze .
I always emphasize structured messages because other messages may not be
easily analyzed .
Over the studies of this system , there may be some modifications in the
base system parts or packages to facilitate a more effective testing .
These requirements will emerge over time .
Thank you very much .
Mehmet Erol Sanliturk
More information about the freebsd-current