[RFC] test layout/standardization for FreeBSD

[Simon J. Gerraty]

On Mon, 19 Nov 2012 08:09:41 +0000, "Poul-Henning Kamp" writes:
>Some tests are not 100% deterministic no matter what.  That is
>an important distinction.

Yes.  It would probably be worth defining some meta tags 
to identify things like that.
Even if the outcome were nothing more than a warning to the user
"hey the results here can vary - don't panic"

Still, to me, tests of that nature start to sound more like system tests
than unit-tests.

>>Also, the estimated duration for tests is very platform specific.
>
>Only if CPU or I/O bound.  Tests for things like dynamic route-expiry,
>and other timeout driven parts of the system will have long realtime
>clock, but short cpu-clock.

Again, that sounds more like a system test than unit-test.
For a unit-test you would want to mock the environment so that your
timer events or whatever are fully controlled (and compressed).

>I fully agree with your "crawl before we run" approach, but testing
>is a quite mature, if underappreciated, discipline, and we would do
>well to learn from others mistakes.

Agreed.  One of the common misstakes it seems is bluring the boundaries
between unit, function and system tests.

System tests are needed for sure, but simple unit-tests (of the bits
that can be simply unit-tested) are far more valuable for establishing
good overal test coverage.

>All I'm asking at this point, is that we make a space for storing
>this metadata in a machine-readable format.

Sounds reasonable, though exactly what and how, is to be determined,
and my hope is that such info should not be needed until we start
pushing unit-tests into system test territory.

Thanks
--sjg