Re: Add jail execution environment support to the FreeBSD test suite

On Thu, Feb 22, 2024 at 9:58 PM <igor.ostapenko@pm.me> wrote:

>
> There is a proposal to improve the FreeBSD test suite.
>
>
>
Thanks a lot for working on improving the parallel tests!

At work, we are using:
1. a Nanobsd based, so a lot of WITHOUT_ : WITHOUT_JAIL, WITHOUT_PF, no
VNET (a nightmare to debug network stack with it), etc.;
2. The standard full test suite (about 8200 tests) is reduced to about 7400
tests with this lighter nanobsd;
3. We have been using kyua parallelism=8 for years, and I had to skip only
about 10-20 tests that were not compliant with parallel mode.

It toke about 23 minutes to run them (in a bhyve VM):

Test cases: 7429 total, 905 skipped, 29 expected failures, 1 broken, 4
failed
Start time: 2024-02-21T00:15:42.527291Z
End time:   2024-02-21T00:38:57.036211Z

Because we were using WITHOUT_JAIL, to improve the time spent, one of my
ideas was to divide the tests into groups of 4 or 8 and run as many bhyve
VMs in parallel.
I assumed that by running regression tests in a jail, I was testing the
jail system as much as the test object (and for our use case, I don’t care
about the jail because we are not using it).
So using the bhyve VM I had the feeling the tests were more confined.

Now if WITH_JAIL becomes mandatory to be able to run the test, I will just
have to modify my build system (like by adding a WITHOUT_JAIL just at
installworld time and not during buildworld).

Regards,
Olivier