Re: RFC: Add required_klds metadata to Kyua

On 6 Nov 2024, at 21:13, Olivier Cochard-Labbé wrote:
> On Wed, Nov 6, 2024 at 7:12 PM Igor Ostapenko <igoro@freebsd.org> 
> wrote:
>
>>
>>
>> The problem to solve here is a well-known one: typically we invoke 
>> "kyua
>> test" and get a lot of "Skipped" due to many kernel module 
>> requirements.
>> And
>> the pain is to know which ones need loading, especially if we target 
>> a
>> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a 
>> new
>> test is added with new module requirements. And other related issues.
>>
>>
>> Hello Igor,
>
> Thanks again for working on this topic
> And there are even more challenging cases: what about tests that 
> assess the
> behavior of different modules and need to unload them as well?
> For example, sys/netipsec/tunnel/* or sys/opencrypto/, which involve
> loading and unloading modules between tests."
>
That’s a good point. I ran into that a few days ago when I looked at 
using the new execenv=jail option to parallelise the netipsec tests, and 
discovered that that wasn’t straightforward.

I wonder if an `forbidden_klds` or something is the answer there. At 
first to skip a test if the forbidden module is present, later on 
perhaps to enforce unloading of the module so the test can run.

That may not be straightforward through, because module state is global, 
and if we only support loading required modules that’s easy (gather 
the list and load, or load when starting a test that requires a new 
module), but if we support unloading them it’ll affect test 
scheduling. (i.e. we can’t load modules while a `forbidden_klds` test 
is running, and can’t unload while a `require_klds` test is running).

That said, perhaps we don’t need to worry about that just yet. It’s 
a minority of tests that forbid modules, so those can just keep doing 
what they’re doing already (i.e. run exclusively, do the checks in the 
test script itself). It shouldn’t stop us from making the majority of 
tests better.

Best regards,
Kristof