Kyua/ATF as a test framework discussion

[Garrett Cooper]

(developers BCCed; please continue discussion on -testing@)
Hi all,

	A few questions came up yesterday at the BSD Vendor Summit in the Q&A session after rodrigc’s talk about Kyua/ATF related to testing on FreeBSD, as related to the testing framework (there are a handful of other questions I’ll bring up in other threads ;)..). In particular, some of the questions that stood out in my mind were:

	1. What type of testing does Kyua do? Can it be used in benchmark testing, stress testing, etc?
	2. Is there a structured way of checking for feature <X> before running my tests?
	3. Can I run tests with custom arguments, e.g. point something at a filesystem path, etc? How do I do that?

	I’ll try to answer these questions based on experience and input I’ve received from several people both internal and external to EMC/Isilon.

Answer to 1.

- Kyua/ATF excels as a unit testing framework and only does non-distributed testing.
- Kyua/ATF should not be used in benchmark testing or stress testing.

	There is a key component missing from kyua that allows it to run functional tests (both on FreeBSD and in general). In particular it does not have per test program setup/cleanup hooks that other frameworks do (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase setup hook. This is something that I called out several years ago when I first looked at importing ATF into the base tree (see the “Known Issues” section in https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup hooks has made some testing at EMC/Isilon painful because of how some testers have written tests, in particular there are testcases that were added to functionally test out mountd, mount_nfs, syslogd, etc that I had to disable because of the unnecessary complexity involved in trying to manage state in setting up/tearing down daemons, running syslogd, etc, and because of the external state involved the testcases would work sometimes if run end-to-end, not other times if executed one-by-one, and not other times if run end-to-end.

	Adding these hooks isn’t impossible and should be the next step in improving the test Kyua/ATF infrastructure (I ran out of time before trying to implement this, and I’m not a C++ whiz so I got lost trying to figure out how to make everything work).

Answer to 2.

	There isn’t currently a structured API for asking questions like “is this driver loaded?", “is this sysctl set?”, “am I running on UFS/ZFS?”, etc. In general test cases from tools/regression, etc have invented homegrown logic for handling this, so this isn’t a new issue.

	I’ve done some initial prototyping of a C/shell library for testing for feature requirements on FreeBSD, but the work isn’t complete/in the tree yet (please see the mailing list threads below for more details):

- https://lists.freebsd.org/pipermail/freebsd-testing/2014-June/000439.html
- https://lists.freebsd.org/pipermail/freebsd-testing/2014-July/000467.html

	I want to complete this sometime before/around the start of the 2015, in particular because this blocks several of the testcases at EMC/Isilon from being upstreamed in a good, working state.

Answer to 3.

	Yes and no(-ish).

For ATF tests:

	You can pass through custom arguments on the command line like so:

	% kyua test -v mykey=myvalue

	This information is that passed through the atf APIs so you can query the value with either atf-config or atf_tc_config (unfortunately the mechanism I’m used to differs enough from what is currently in place, so I’ll need to go look up what the current mechanism/structure is for this, but basically it involves specifying variables kyua.conf).

For non-ATF tests:

	You have to resort to using environment variables today, like I did with pjdfstest: https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest/tests/conf?revision=274016&view=markup#l5 , as I described in the README file: https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?revision=274016&view=markup . I’d _really_ like to avoid doing this though because then we’ll have 40 some different ways to do things around the tree instead of a single, supported way to do things.

	There was an enhancement I opened on GitHub to add “configuration integration” for the other test types (Plain, TAP) [via environment variables or command querying], but that wasn’t completed: https://github.com/jmmv/kyua/issues/84 . I’ll reopen the discussion.

	I’ll try to add/migrate relevant links for things noted in this email on the wiki.

Thank you!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-testing/attachments/20141104/43b8a3d7/attachment.sig>