Kyua/ATF as a test framework discussion

[Julio Merino]

On Wed, Nov 5, 2014 at 5:02 PM, Craig Rodrigues <rodrigc at freebsd.org> wrote:
>
>
> On Wed, Nov 5, 2014 at 1:41 PM, Julio Merino <jmmv at freebsd.org> wrote:
>>
>> On Tue, Nov 4, 2014 at 2:40 PM, Craig Rodrigues <rodrigc at freebsd.org>
>> wrote:
>> > Having hooks in the test framework to do setup/teardown like in other
>> > unit
>> > test frameworks would be useful.
>>
>> Please tell me of a test framework that has setup/teardown
>> functionality outside of the test program. I would like to see how
>> they handle it.
>
> http://nose.readthedocs.org/en/latest/writing_tests.html#test-functions
> https://docs.python.org/3/library/unittest.html#unittest.TestCase.setUp
> http://cunit.sourceforge.net/doc/managing_tests.html
> http://junit.sourceforge.net/junit3.8.1/javadoc/junit/framework/TestCase.html#method_summary

All of these examples point at *in-program* setup/teardown methods,
not what you were describing a while ago with hooks in the Kyuafile.

Adding per-test case setup/teardown methods to ATF would be easy, but
mostly pointless except for maybe in atf-c++ (but I don't think anyone
cares about atf-c++).

Adding per-test program setup/teardown methods to ATF is interesting,
but difficult due to the semantics of the interface of the test
programs. Also, the resulting API would be quite hard to handle and
understand from a programmer's perspective...

See my other mail in this thread. Basically, I believe the current ATF
interface is not worth fixing to support these cases and we should
look at other existing, unit-testing libraries to see if they fit our
needs. Kyua should be the glue to bring them together. In fact, you
should be able to use all the libraries you mentioned with Kyua
already (or at least that's the goal); and, if not, that's a bug that
we should fix.