[RFC] test layout/standardization for FreeBSD

[Garrett Cooper]

On Wed, Nov 14, 2012 at 1:43 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>wrote:

> --------
> In message <CAGH67wSoU08N8=
> 40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg at mail.gmail.com>
> , Garrett Cooper writes:
>
> >I asked for feedback from some of the stakeholders of
> >the FreeBSD test automation effort [...]
>
> Can I throw in something here ?
>
> A very important thing is to have systematic metadata about test-cases,
> in particular:
>
> A) Is the test 100% deterministic, or is there a risk of random failure ?
>
> B) Is the test reliable on a heavily loaded machine ?
>
> C) Estimated duration of test
>
> etc.
>

- These are important points and to be clear based on discussion prior to
and up to the vendor summit, the goal was to provide deterministic
unittests for the first prototype (which covers point A and implicitly
point B); the goal from an end-user perspective as to how they would be
affected today is that unless make test[kernel,world] was actually called,
it would just built and install the tests as necessary
- Point C cannot be accurately answered as it depends on what tests are are
in a tree, run via `make test[kernel,world]`, etc.

(going into B) further) Other items such as fault tolerance, stress, etc
need to be worked into the overall test ecosystem of FreeBSD (I recognize
the need for integrating pho's stress2 project into FreeBSD proper so it's
run on a regular basis, the need for adding to stress2, etc), but it wasn't
in the scope of the initial work being done for the ATF FreeBSD port / in
the short to medium term for test automation on FreeBSD and as such hasn't
been planned in the overall "automation roadmap".

Going further, as far as items B/C are concerned I would argue that they
are highly dependent upon the test matter, how the testcases are
implemented, etc. From my perspective the goal is to have (from a
functional perspective) simple testcases that exercise functionality to
ensure that requirements noted in manpages (because that's the user facing
documentation) are adhered to and not regressed, and then move on to other
more critical items requiring attention (again, stress, fault tolerance,
etc). Less deterministic items need to be factored out of standard runs
and/or decorated (like how one can do in python) such that they aren't
necessarily executed by default, but rather when explicitly requested,
either as individual tests or as a group of tests.

As the phrase goes, "Rome wasn't built in a day".

Thanks,
-Garrett