Re: RFC: Add required_klds metadata to Kyua

On Wed, Nov 6, 2024, 10:12 AM Igor Ostapenko <igoro@freebsd.org> wrote:

> Hi FreeBSD developers,
>
> During Kyua's execenv=jail feature implementation there were discussions to
> get some help from Kyua side with kernel modules required by specific
> tests.
> The thing got "required_klds" codename.
>
> The problem to solve here is a well-known one: typically we invoke "kyua
> test" and get a lot of "Skipped" due to many kernel module requirements.
> And
> the pain is to know which ones need loading, especially if we target a
> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new
> test is added with new module requirements. And other related issues.
>
> I would like to share the existing ideas and vision before the
> implementation itself. I propose to split it onto several subsequent
> projects to ease reasoning, review, etc.
>
>
> Project A: declaration & skipping
> ---------------------------------
>
> Add a new metadata property to Kyua which is used by a test to define its
> requirements on specific kernel modules expected being loaded. The
> following
> interface is expected:
>
> - Define it on ATF level, e.g. for atf-sh:
>
>      atf_set require.klds pf pflog
>
> - Define it on Kyuafile level:
>
>      test_program{name="testbinary1", required_klds="pf pflog"}
>
> - Get it listed per test case:
>
>      > kyua list -v testbinary1 | grep required_klds
>      required_klds = pf pflog if_epair
>
> Having it declared we could simplify tests and avoid manual checks for
> modules with the respective "skip test" lib calls. Thus, this new metadata
> would work the same way as the existing required_programs and
> required_files
> ones -- if some of the required modules are not loaded Kyua will skip such
> test with a respective message including names of the missing modules.
>
> An extra benefit is that we do not need to iterate it like: 1) kyua test,
> 2)
> kldload a-missing-module, 3) repeat. Such iteration is usually needed due
> to
> our tests typically do kldstat sequentially with "skip fast" logic.
> Skipping
> reason information still is not aggregated to make it easy to run a single
> kldload for all missing modules, but that's another story which is covered
> by the following Project B.
>
> Q1: I have doubts regarding the name of this new metadata. There are
> several
> other options in my mind, it would be appreciated to hear opinions
> regarding
> this:
>    a) required_klds -- the original idea based on the vibe that FreeBSD
> Kyua
>       fork is mainly for FreeBSD
>
>    b) required_mods -- to keep it OS agnostic while Kyua still is treated
>       as an OS agnostic tool, and freebsd/kyua GitHub repo actually is the
>       main Kyua repo in the world today
>
>    c) required_kmods -- a little improvement of the previous option to make
>       sure it talks about "kernel modules" and leave space for other
> meanings
>       in the future, just in case
>
>    d) required_klds | required_kmods -- to have some official aliases
>       supported, but it's expected to bring confusion instead
>
>    e) your variant
>

Love this. Would go with a FreeBSD name, though, because no two oses have
the same module names. Don't care a huge amount.

Would also like an option to never load kld so qemu testing can be a lot
faster...

Warner



> Project B: automatic kldload
> ----------------------------
>
> Having Project A implemented and the existing tests augmented with the
> respective metadata we could ask Kyua to automatically load the required
> kernel modules.
>
>  From the implementation perspective, Kyua is expected to have a generic
> concept of a requirement resolver. A resolver may read all metadata and
> decide on its actions. If we decide on "klds" name in Project A then the
> very first resolver implemented would look for required_klds metadata, if
> it's present, then the mentioned kernel modules will be loaded. Such
> resolver could be named "klds", and we can think of future additions like
> "pkgs" resolver and so on.
>
> Currently, I see it as a new kyua command to make it separate from the
> testing process itself. The interface could be as follows:
>
> - List all available requirement resolvers (name and short description):
>
>      > kyua rr list
>
> - Run all resolvers or only specified ones. It would deal with the current
>    dir, and later we can think of "-k" option like "kyua test" command has.
>
>      > kyua rr run [resolver1-name resolver2-name ...]
>
> - It's proposed to be a sub-set of commands in case if in the future we
> want
>    additions, e.g. dry-run or verify command without actual resolving.
>
> Having this interface the FreeBSD CI could have a separate step to run
> "kyua
> rr run" before the existing step which runs "kyua test". But from
> development perspective it may quickly end up with a need to have a single
> command for both. The following interface addition could be there:
>
> - Run all or specified requirement resolvers before the actual testing:
>
>      > kyua test --rr ["resolver1-name resolver2-name ..."]
>
> Another option is to use the existing Kyua mechanisms and add it as a
> configuration variable, i.e. it can be configured once in the local
> kyua.conf not to specify it with every test command invocation:
>
>      # without configuration:
>      > kyua -v rr="klds pkgs" test
>
>      # with configuration:
>      > echo 'rr = "klds pkgs"' >> /etc/kyua/kyua.conf
>      > kyua test  # now it does resolving first
>
>
> Q2: Project B has the same set of questions about naming and style. For
> example, "rr" name for kyua.conf feels non-obvious and probably it could be
> named like "requirement_resolvers" to keep it self-explainable.
>
>
> What do you think? Any idea, comment, or suggestion are welcome.
>
>
> Best regards,
> igoro
>
>