Re: Add jail execution environment support to the FreeBSD test suite

On Friday, February 23rd, 2024 at 11:41 AM, Olivier Cochard-Labbé <olivier@freebsd.org> wrote:
> 
> Thanks a lot for working on improving the parallel tests!
> 
> At work, we are using:
> 1. a Nanobsd based, so a lot of WITHOUT_ : WITHOUT_JAIL, WITHOUT_PF, no VNET (a nightmare to debug network stack with it), etc.;
> 2. The standard full test suite (about 8200 tests) is reduced to about 7400 tests with this lighter nanobsd;
> 3. We have been using kyua parallelism=8 for years, and I had to skip only about 10-20 tests that were not compliant with parallel mode.
> 
> It toke about 23 minutes to run them (in a bhyve VM):
> 
> Test cases: 7429 total, 905 skipped, 29 expected failures, 1 broken, 4 failed
> Start time: 2024-02-21T00:15:42.527291Z
> End time: 2024-02-21T00:38:57.036211Z
> 
> Because we were using WITHOUT_JAIL, to improve the time spent, one of my ideas was to divide the tests into groups of 4 or 8 and run as many bhyve VMs in parallel.
> I assumed that by running regression tests in a jail, I was testing the jail system as much as the test object (and for our use case, I don’t care about the jail because we are not using it).
> So using the bhyve VM I had the feeling the tests were more confined.
> 
> Now if WITH_JAIL becomes mandatory to be able to run the test, I will just have to modify my build system (like by adding a WITHOUT_JAIL just at installworld time and not during buildworld).
> 


Thanks for your consideration.

Actually, from the very beginning I've had doubts regarding a case when jail
feature is not present, but some of existing tests which freely use jail w/o
additional things like "require.progs jail" and absence of comments regarding
that during the initial review of the patch formed a feeling that it does not
require effort to consider, at least for now. Thanks for mentioning a real
production case, it helps not to guess its priority.

From a glance it looks like the following tasks could be added to TODO:
- kyua can be built WITHOUT_JAIL
- kyua should detect in runtime which execenv it supports, host only or both
- kyua should skip tests which are not supported by a kyua runtime instance

It looks to cover your case. It would need no changes on your side,
execenv="jail" based tests would simply be skipped, the similar way when we
run amd64 tests on aarch64, and so on.


Could you please share your src config (WITHOUT_*, etc)?


Best regards, Igor.