ATF config variables for FreeBSD tests

[Alan Somers]

On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino <jmmv at freebsd.org> wrote:
> On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers <asomers at freebsd.org> wrote:
>> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino <jmmv at freebsd.org> wrote:
> [...]
>>> I think these all sound reasonable.
>>>
>>> Can these expected side-effects be reversed in the test cleanup?
>>
>> For the first two, yes.  I doubt that Kyua could do it automatically,
>> but each test case certainly can.
>
> Right. I do not think Kyua should get into these tricky details
> either, if only because they are too OS-specific. But we can certainly
> offer additional scripts/functions (where it makes sense) in the
> FreeBSD test suite to simplify the test cases that might need this.

Definitely.  This is a good case for a library of reusable test code.
For example, I have two separate test programs that would like to
share setup and cleanup code for FIBs.  Should I put it in
/usr/tests/include/fibs.sh or /usr/tests/lib/fibs.sh ?  I don't like
those options, because it looks like fibs.sh is a test program
designed to test stuff in /include or /lib.  Perhaps
/usr/tests/tests/sh/include/fibs.sh?  I'm really not sure where the
best place would be.  What do you think?

>
>>  Perhaps even for the fourth, though
>> it would be tricky.  But there's no way to reverse the effects for the
>> third.  The ZFS tests, for example, create and destroy zpools.  How
>> would you reverse that?  You can't.  Whatever was previously on the
>> disk is gone.
>
> Ah, I guess I missed that detail.
>
> Couldn't those tests run on top of vnd devices though?

FreeBSD doesn't have vnd(4), though it has md(4), which is similar.
md(4) devices could be used for some ZFS tests.  as it happens, ZFS
can also use file-backed vdevs.  But these workarounds don't work in
all cases.  Sometimes you need real disks:
1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've
one a lot of work.
2) Neither file-backed vdevs nor md(4) devices have physical path
information, which is needed to test some hotspare functionality.
Only da(4) devices that are attached to ses(4) expanders have that.
3) There is no way to remove an active file-backed vdev, and I don't
think that you can destroy an in-use md(4) device either.  Therefore,
to test how ZFS handles drive removals requires real drives.
4) gibbs has made many tweaks to ZFS to better support 4K and 8K
sector drives.  There is no way to emulate those with file-backed
vdevs.  Perhaps it could be done with md(4) and gnop(8), but I've
never tried.
5) pjd and Steve Hartland have been working on TRIM support.  That's
not supported by file-backed vdevs, and I doubt that md(4) supports it
either.
6) Copy-on-write on top of copy-on-write is very slow.  If your system
uses ZFS root, then both file-backed vdevs and file-backed md(4)
devices are doing COW-on-COW.  Using physical disks for the tests
greatly speeds the tests' runtimes.

Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is
defined, and use file-backed vdevs otherwise.  Tests that absolutely
require physical disks would be skipped if test_suites.FreeBSD.disks
isn't defined.  It would take some work, but it's doable.

>
> [...]
>>> And lastly, we'd just need a simple "filtering" feature in the kyua
>>> cli to allow specifying which size of tests to run (or to filter by
>>> any other metadata property, for that matter).
>>
>> I don't like this idea.
>
> I hope you are objecting to the filtering by test sizes, not the
> filtering itself!  See below.
>
>>  We tried it at work, and it didn't work out
>> very well.  Basically, I automatically assigned sizes (short, medium,
>> long) to all of our tests based on their runtimes, so users could
>> select to only run the short or medium tests on the bench.  The
>> problem is that the classification didn't make sense.  For an
>> expedited test run, you don't want the shortest tests; what you want
>> are the tests with the most value per unit time.  Unit tests have a
>> high value and a very short runtime.  Stress tests have a very long
>> runtime and arguably low value since they don't always have consistent
>> results.
>>
>> At my previous job there was a large department whose duties included
>> curating test suites and deciding which tests would be included in the
>> short runs.  FreeBSD isn't going to have that, but we could still ask
>> test authors to classify tests along the lines that I suggested.
>
> That's a good point. If I understand you correctly, what is important
> is to manually curate the set of "smoke tests" that run very quickly
> and that provide a reasonably high assurance that major subsystems
> haven't broken.
>
> We can do this too pretty easily, but configuration variables are the
> wrong mechanism. We can already define this information in metadata
> properties as long as you prefix them with X-. So all we'd need is a
> way to tell Kyua to only run tests marked with such property (i.e.
> with the filtering feature described above).

I was assuming that a test case could put in its header
'require.config "stress_tests"', or something like that.  Then runtime
filtering would work by which variables the user defines.  That way,
no new work would be required in Kyua.  But it sounds like you're
talking about something different.  IIUC, you're suggesting that a
test case would say in its header 'atf_set "X-runtime" "stress_test"'
and then some TBD filtering mechanism in Kyua selects on it.  That
would certainly be more intuitive.  Is that what you had in mind?

-Alan