FreeBSD Testing Facility

[Mehmet Erol Sanliturk]

On Sun, Feb 24, 2013 at 12:48 PM, Alexander Yerenkow <yerenkow at gmail.com>wrote:

> How about create -testing mail list, and start searching for any kind of
> volunteers?
> I could lend a hand in creating auto-testing some aspects of  FreeBSD (at
> leas successful booting/working net/route etc),
> someone else probably could have to say something too.
>
> I think resources are there, they just need to be gathered :)
>
> --
> Regards,
> Alexander Yerenkow




Really , there are ample amount of facilities in FreeBSD already .
The steps may be the following :

- Prepare a testing requirements document :
    There are existing testing programs in the svn .List them .
   . The developers may use tests that are not in svn . Collect information
about them .
   . Collect additional testing point suggestions .

. Design a testing facility :

  This may be just an imitation of the packages/ports tree .

  For each test , prepare a package .

  The new package management project contains very important information to
imitate .
  Prepare a test_apply program just like pkg_add .

  Generate an ftp site parts for .../testing/Fatal/... ,
.../testing/Dangerous/... , .../testing/Harmless/...

  Generate databases for the collected result mails , with respect to
message groups .

  For each test , generate an ( xml , or JSON , or just  name = value pairs
( like .conf files ) ) results
  e-mail structure .
  During tests , this file will be created .

  When such an e-mail is received into a definite address , with program ,
introduce it into a database .

 Generate reports from databases to submit to developers .

 Generate a mailing list about testing facility .

 Prepare e-mail templates with respect to testing points ( groups ,
subjects ) .

To the subscriber , present a form to fill information which parts he or
she can test , and store these into a data base .

 Associate each test package with fields of the subscribers testing
possibilities .

 When a testing action is required ( generated by a modification in a
tested area ) , query the subscribers database , and send an e-mail to
possible testers .

Before test_apply program is executed , if it is necessary to perform some
actions , prepare an algorithm to define such steps and attach to e-mail ,
or fetch by test_apply .

Implementation :

The above steps may be implemented mostly by facilities present in the base
system and packages in the ports tree .

Identify required new programs , write-ups , scripts , etc. , to design ,
and implement them .

Application , Testing :

By using a harmless test , apply the above designed system and evaluate
outcome ,
recycle the above steps ,
Start to populate the testing system with packages one by one .

Maintain the testing tree , and when a modification is applied into a part
, if necessary , also
modify the testing package and request a testing step .


Now , there are "Call for Testing"  applications in the present mailing
lists .
With the above structure , such requests will be more widely applied and
will produce structured ,
analyzable results .

If one such package is developed , the rest will start to imitate its many
steps , development of new testing packages will be easy .


Some of the test packages may also be installed during a complete FreeBSD
installation and may be used to test the installed system , and the results
may be collected to analyze .


I always emphasize structured messages because other messages may not be
easily analyzed .


Over the studies of this system , there may be some modifications in the
base system parts or packages to facilitate a more effective testing .
These requirements will emerge over time .


Thank you very much .


Mehmet Erol Sanliturk