Skipping tests that are unimplemented in 32-bit emulation

[Alan Somers]

On Tue, Aug 7, 2018 at 10:34 AM, Mark Johnston <markj at freebsd.org> wrote:

> On Tue, Aug 07, 2018 at 10:11:45AM -0600, Alan Somers wrote:
> > On Tue, Aug 7, 2018 at 9:24 AM, Mark Johnston <markj at freebsd.org> wrote:
> > > Oops, right.  For 2), then, couldn't we define test_suite("FreeBSD32")
> > > in the kyuafiles for the 32-bit test suite and set a
> > > test_suite.FreeBSD32.compat32=true variable in kyua.conf?  Then a test
> > > which uses, say, setfib(1) can query that variable and skip if it's
> set.
> > >
> >
> > Ok, but what will set the test_suite.FreeBSD32.compat32 variable?  It
> can't
> > be set as part of the image build, because then it will have the wrong
> > value in jails and chroots.  It needs to be set at runtime.
>
> For my suggestion to work, the 32-bit test suite must be built with
> TESTSUITE=FreeBSD32 instead of the default in suite.test.mk.  This could
> be accomplished in a few ways:
> 1) Modify the amd64 build to additionally build a 32-bit copy of the
>    test suite with TESTSUITE=FreeBSD32 and install it to /usr/tests32.
>    Users that wish to run it would have to copy it into their
>    chroot/jail.
>
2) When building a 32-bit image for a chroot/jail, just remember to
>    specify TESTSUITE=FreeBSD32 in the make(1) invocation.  (I didn't
>    actually verify that this does what I expect.)
> 3) Always define the test suite name to be FreeBSD32 or FreeBSD64
>    depending on the target architecture.  Then, tests that need to be
>    skipped when run under COMPAT32 would need to check for the
>    "compat32" variable _and_ determine if the host kernel is 64-bit
>    (e.g., by looking at the hw.machine sysctl).
>
> If you do 2) and just set test_suite.FreeBSD32.compat32 statically in
> /usr/local/etc/kyua.conf, then I don't see why you need any special
> runtime logic.  Any tests that aren't supported by COMPAT32 can just be
> skipped when the compat32 kyua variable is set.
>

Option 2) may avoid additional runtime logic, but at the expense of
creating an image that won't work correctly when run natively.  Plus, you
would also have to copy all of test_suite.FreeBSD's Kyua variables to
test_suite.FreeBSD32.  For example, test_suites.FreeBSD.disks.

I still think the simplest option is to build a single test suite, where
the few tests that can't run under emulation will call a helper function
that compares the build architecture to the current kernel and skip on a
mismatch.  It eliminates extra configuration, uses just a single image, and
doesn't modify a large number of files.

Are there any emulations besides amd64/i386 that we support?  Like
mips64/mips for example?

-Alan