Re: Add jail execution environment support to the FreeBSD test suite

On Mon, Feb 26, 2024 at 12:58 PM Igor Ostapenko <igor.ostapenko@pm.me>
wrote:

> On Friday, February 23rd, 2024 at 6:30 PM, Brooks Davis <
> brooks@freebsd.org> wrote:
> >
> > > 2 The Idea
> > >
> > > The idea is not new. A test could be running in a jail -- it provides
> the
> > > required isolation with minimum or zero effort from a test.
> >
> >
> > This generally sounds good. One minor concern I have is how this would
> > interact with the ability to run the test suite in a jail. This is
> > imperfectly supported today (IIRC ~350 failures on amd64), but it's
> > quite userful for testing sweeping userspace-only tests like libsys and
> > I'd love to see support expanded and improved (failures fixed or tests
> > skipped, poudriere jail support, etc).
> >
>
> Thanks for your consideration.
>
> As I understand it's a question to the specific tests, which needs revising
> and tuning for a non-prison0 case. I saw such issues with existing tests,
> frequently it was about jail restrictions. And by design, a root within a
> jail
> cannot alter restrictions of its parent jail, i.e. it's up to the creator
> of
> such jail. In contrast, execenv=jail asks Kyua to create a temporary jail
> for
> a test execution, and Kyua can be asked to configure it for the needs of a
> test via execenv_jail. But when Kyua itself is running within an existing
> jail
> then expectations of the test suite should be lifted a level above,
> especially for usual execenv=host based tests. Thus, it's out of Kyua's
> control to help with that. And this proposal does not change or improve
> this
> case.
>
> Just a quick idea here. Probably, the test suite could help a creator of
> such
> jail above Kyua itself with its metadata. For example, a test is not
> intended
> to be run within a jail and it works fine if Kyua runs within prison0, but
> if
> Kyua itself runs in a jail it starts failing due to missing, for instance,
> allow.mlock allowence, see jail(8). A test could define
> execenv_jail="allow.mlock" metadata without asking for execenv=jail, i.e.
> it's
> like a hint for a non-prison0 case. And Kyua could aggregate and provide
> such
> information to ease creation of a jail above the Kyua itself.
>
> On the other end of the spectrum, we could think of a new feature of
> jail(8)
> to be able to ask it for a jail with maximum possible set of allowed
> things.
> Just thinking out loud.
>

So I run the test suite in jail too today. To test how well qemu user-mode
stuff
is working. I see closer to 800 failures IIRC... Some of these are easy
(like
all the geom tests that need /dev/md.ctl to create the md they test with),
but
others are harder and whack-a-mole: mknode tests, for example, will always
fail (at least with the default jail settings) and so should be fixed.
There's several
tests that would work in a native jail that would fail in a qemu jail
because the
system calls are missing, of which I think the jail system calls are in the
list...

I also run it on amd64 machines in a i386 jail and on aarch64 machines with
a armv7
jail too. But I've done it a lot less, and don't have good numbers for
doing that. I
suspect they will be slightly more than Brooks is seeing since our 32-bit
emulation
is imperfect.

So long as we keep the concepts of 'run this in the current env vs creating
a new jail'
and 'test expected to fail inside a jail vs work in un-jailed proc'
separate, I think
we'll be fine. And it might be nice to also have a 'don't run tests that
need temp jails'
option as well that I could mash for times I know they will fail to even
launch.

Cool stuff.

Warner