Re: RFC: Add required_klds metadata to Kyua

On Wed, Nov 6, 2024 at 1:27 PM Kristof Provost <kp@freebsd.org> wrote:
>
> On 6 Nov 2024, at 21:13, Olivier Cochard-Labbé wrote:
>
> On Wed, Nov 6, 2024 at 7:12 PM Igor Ostapenko <igoro@freebsd.org> wrote:
>
> The problem to solve here is a well-known one: typically we invoke "kyua
> test" and get a lot of "Skipped" due to many kernel module requirements.
> And
> the pain is to know which ones need loading, especially if we target a
> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new
> test is added with new module requirements. And other related issues.
>
> Hello Igor,
>
> Thanks again for working on this topic
> And there are even more challenging cases: what about tests that assess the
> behavior of different modules and need to unload them as well?
> For example, sys/netipsec/tunnel/* or sys/opencrypto/, which involve
> loading and unloading modules between tests."
>
>
> That’s a good point. I ran into that a few days ago when I looked at using the new execenv=jail option to parallelise the netipsec tests, and discovered that that wasn’t straightforward.
>
> I wonder if an forbidden_klds or something is the answer there. At first to skip a test if the forbidden module is present, later on perhaps to enforce unloading of the module so the test can run.
>
> That may not be straightforward through, because module state is global, and if we only support loading required modules that’s easy (gather the list and load, or load when starting a test that requires a new module), but if we support unloading them it’ll affect test scheduling. (i.e. we can’t load modules while a forbidden_klds test is running, and can’t unload while a require_klds test is running).
>
> That said, perhaps we don’t need to worry about that just yet. It’s a minority of tests that forbid modules, so those can just keep doing what they’re doing already (i.e. run exclusively, do the checks in the test script itself). It shouldn’t stop us from making the majority of tests better.
>
> Best regards,
> Kristof

I too like the idea of Project A with required_klds or required_kmods.
But how would you handle situations where a user customizes their
kernel config to build some feature that's usually a module directly
into the kernel?  I would think that would break any test using
required_klds.

But project B is going to be a lot harder.  "forbidden_klds", as
suggested by kp, might be the way to go.  In addition to the
abovementioned suites, the fusefs tests are incompatible with the
mac_bsdextended kld. I suggest focusing on Project A for now.  That
would be great even by itself.

-Alan