Panicking UUTs and integrating the ZFS test suite into the default run

[Garrett Cooper]

(moving to -testing and renaming because this is turning into a bikeshed discussion)

On Oct 25, 2014, at 9:49, Craig Rodrigues <rodrigc at freebsd.org> wrote:

> On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein <alfred at freebsd.org> wrote:

...

> If tools/regression/zfs has Kyuafiles that makes it easier to
> run under the kyua tool, that would greatly facilitate running
> under Jenkins automation, and would be useful.
> I notice some of the fixes you are applying to the regression/zfs
> tests is to make certain tests not run on FreeBSD because they
> cause known kernel panics such as this:
> 
> https://lists.freebsd.org/pipermail/svn-src-all/2014-October/093671.html

Yes.

> I'm not entirely convinced that this is a good way to "fix" a test.
> If a test causes a panic, then that's what it does, it it should not be
> swept under the rug, as Alfred has pointed out.
> Printing out a warning with a pointer to the PR just before running this
> type of test is OK, though.

I highly disagree. Neither of us can guarantee that these tests aren’t being run on either production systems (this would be extremely foolhardy, but I’ve seen it happen before several times on the LTP list), test systems that other developers/testers are working on where the focus is not ZFS, or that the end-user would care about ZFS in their test run. Continually panicking the box because of known issues is silly and wastes developer/tester cycles. Furthermore, running the tests and getting a subset of coverage [because the box panicked] doesn’t help further the goals of testing the overall system. Good developers and testers care about coverage in addition to “how green are my tests?”, “how far is feature X to completion?”.

Idioms like the following are in my opinion sufficient for working around issues.

  6 
  7 [ "${os}" = "FreeBSD" ] && die "panics FreeBSD; see bug # 194589"
  8

1. The issue is documented, so developers and testers know where to get the bug information
2. The test is marked as a failure.

Both of these items encourage testers and other developers to go prod developers with knowledge and cycles to go fix the bugs so things eventually turn green.

> I agree with Alfred on this.  Even though Alan's test suite
> may kernel panic or cause problems, there is still value in running it
> and making the results visible on jenkins.freebsd.org.
> Running these tests inside a VM which is generated during the build
> will allow these types of test to run, but still keep the test machine usable,
> even if the VM gets corrupted while running the tests.

Again, because of some of the things that Alan said about corruption and panicking, until the underlying issues are fixed in ZFS, there’s absolutely no reason why it should be run by default.

As a sidenote, the FreeBSD test suite needs a couple features:
1. Additional configuration options, similar to “allow_sysctl_side_effects” used by tests/sys/kern/kern_descrip_test.c (say “causes_corruption” or “might_panic”). If
	a. These safety belts existed
	b. Were off by default
	c. Could be queried via all of the handlers (ATF, TAP, Plain) via some API (atf_config, etc).
    then I wouldn't have an issue with someone integrating them in to the Jenkins runs without the bail outs. 1. and 2. are easy to take care of, but 3. is missing in all but the ATF handler.
2. A hook to say “this feature/module is present”, e.g. “this test requires zfs, so if it’s not kldloaded/statically compiled into the kernel, don’t run this test”

> If we have test suites for ZFS, but no one runs them, then no one will
> bother to investigate and fix the bugs.

I understand but someone needs to go fix the bugs too before stuff is turned on by default, otherwise that person will bear the brunt of the support issues that come up with Jane/Joe user decides to go and run the test suite and it goes and blows up their machine.

> Running the test suites under automation that is visible on jenkins.freebsd.org is going to force developers to see problems
> much sooner than they do now.  Just having the tests in the tree
> and hoping that people run them and care to look into the problems is
> not enough.

Cheers.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-testing/attachments/20141025/bbd7d7e6/attachment.sig>