Re: Add jail execution environment support to the FreeBSD test suite

On Friday, February 23rd, 2024 at 11:41 AM, Olivier Cochard-Labbé <olivier@freebsd.org> wrote:
> 
> On Thu, Feb 22, 2024 at 9:58 PM <igor.ostapenko@pm.me> wrote:
> 
> > 
> > There is a proposal to improve the FreeBSD test suite.
> > 
> 
> 
> Thanks a lot for working on improving the parallel tests!
> 
> At work, we are using:
> 1. a Nanobsd based, so a lot of WITHOUT_ : WITHOUT_JAIL, WITHOUT_PF, no VNET (a nightmare to debug network stack with it), etc.;
> 2. The standard full test suite (about 8200 tests) is reduced to about 7400 tests with this lighter nanobsd;
> 3. We have been using kyua parallelism=8 for years, and I had to skip only about 10-20 tests that were not compliant with parallel mode.
> 
> It toke about 23 minutes to run them (in a bhyve VM):
> 
> Test cases: 7429 total, 905 skipped, 29 expected failures, 1 broken, 4 failed
> Start time: 2024-02-21T00:15:42.527291Z
> End time: 2024-02-21T00:38:57.036211Z
> 
> Because we were using WITHOUT_JAIL, to improve the time spent, one of my ideas was to divide the tests into groups of 4 or 8 and run as many bhyve VMs in parallel.
> I assumed that by running regression tests in a jail, I was testing the jail system as much as the test object (and for our use case, I don’t care about the jail because we are not using it).
> So using the bhyve VM I had the feeling the tests were more confined.
> 
> Now if WITH_JAIL becomes mandatory to be able to run the test, I will just have to modify my build system (like by adding a WITHOUT_JAIL just at installworld time and not during buildworld).
> 

It would be great if you can test the patch.
The general test plan instructions are here: https://reviews.freebsd.org/D42350.
It includes instructions for your case WITHOUT_JAIL. It's expected that you
need no changes in your build system, the Kyua should be built without jail
support and your usual test runs should work as usually. And any execenv=jail
based test on your way should be automatically skipped by such instance of Kyua.


Best regards, Igor.