[RFC] test layout/standardization for FreeBSD

[Poul-Henning Kamp]

--------
In message <CAGH67wRFyFrkZWxGEQBA+Y7e1v7JQoFiQyEp0w2qzrtU4KWuyQ at mail.gmail.com>
, Garrett Cooper writes:

>> A very important thing is to have systematic metadata about test-cases,
>> in particular:
>>
>> A) Is the test 100% deterministic, or is there a risk of random failure ?
>>
>> B) Is the test reliable on a heavily loaded machine ?
>>
>> C) Estimated duration of test

>- These are important points and to be clear based on discussion prior to
>and up to the vendor summit, the goal was to provide deterministic
>unittests for the first prototype [...]

That's very laudable, but deterministic tests are not always possible,
and therefore your design should be able to handle non-deterministic tests
even if you are going to frown upon them for all eternity.

>- Point C cannot be accurately answered as it depends on what tests are are
>in a tree, run via `make test[kernel,world]`, etc.

Point C is very important, because if you want to leave space for
comprehensive tests, you will have tests that take many minutes to
execute, and again, you may not like them, but please make space for
them in your design.

>(going into B) further) Other items such as fault tolerance, stress, etc
>need to be worked into the overall test ecosystem of FreeBSD [...]

That is not what I'm talking about here, I'm talking about recording
for each and every unit-test if it can be expected to work on a system
under high load or not.

The point I am trying to drive home here, is that a test-case without
metadata as to its behaviour is not a good idea.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.