Re: RFC: Add required_klds metadata to Kyua

Hi,

The Project B patch is open for testing and discussion:

	https://reviews.freebsd.org/D48087


The outline of the currently proposed UI/UX is as follows:

# kyua prepare
all                     Run all requirement resolvers
kmods                   Resolve required_kmods for FreeBSD

# grep -r kmods *
Kyuafile:atf_test_program{name="divert-to", ... required_kmods="if_epair if_bridge pf ipdivert"}
mbuf:   atf_set require.kmods if_epair if_bridge pf dummymbuf

# kyua prepare --dry-run all
kldload dummymbuf if_bridge if_epair ipdivert pf

# kyua prepare kmods
kldload dummymbuf if_bridge if_epair ipdivert pf



Testing and sharing your thoughts are welcome.


Best regards,
igoro


Igor Ostapenko wrote on 11/6/24 8:12 PM:
> Hi FreeBSD developers,
> 
> During Kyua's execenv=jail feature implementation there were discussions to
> get some help from Kyua side with kernel modules required by specific tests.
> The thing got "required_klds" codename.
> 
> The problem to solve here is a well-known one: typically we invoke "kyua
> test" and get a lot of "Skipped" due to many kernel module requirements. And
> the pain is to know which ones need loading, especially if we target a
> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new
> test is added with new module requirements. And other related issues.
> 
> I would like to share the existing ideas and vision before the
> implementation itself. I propose to split it onto several subsequent
> projects to ease reasoning, review, etc.
> 
> 
> Project A: declaration & skipping
> ---------------------------------
> 
> Add a new metadata property to Kyua which is used by a test to define its
> requirements on specific kernel modules expected being loaded. The following
> interface is expected:
> 
> - Define it on ATF level, e.g. for atf-sh:
> 
>       atf_set require.klds pf pflog
> 
> - Define it on Kyuafile level:
> 
>       test_program{name="testbinary1", required_klds="pf pflog"}
> 
> - Get it listed per test case:
> 
>       > kyua list -v testbinary1 | grep required_klds
>       required_klds = pf pflog if_epair
> 
> Having it declared we could simplify tests and avoid manual checks for
> modules with the respective "skip test" lib calls. Thus, this new metadata
> would work the same way as the existing required_programs and required_files
> ones -- if some of the required modules are not loaded Kyua will skip such
> test with a respective message including names of the missing modules.
> 
> An extra benefit is that we do not need to iterate it like: 1) kyua test, 2)
> kldload a-missing-module, 3) repeat. Such iteration is usually needed due to
> our tests typically do kldstat sequentially with "skip fast" logic. Skipping
> reason information still is not aggregated to make it easy to run a single
> kldload for all missing modules, but that's another story which is covered
> by the following Project B.
> 
> Q1: I have doubts regarding the name of this new metadata. There are several
> other options in my mind, it would be appreciated to hear opinions regarding
> this:
>     a) required_klds -- the original idea based on the vibe that FreeBSD Kyua
>        fork is mainly for FreeBSD
> 
>     b) required_mods -- to keep it OS agnostic while Kyua still is treated
>        as an OS agnostic tool, and freebsd/kyua GitHub repo actually is the
>        main Kyua repo in the world today
> 
>     c) required_kmods -- a little improvement of the previous option to make
>        sure it talks about "kernel modules" and leave space for other meanings
>        in the future, just in case
> 
>     d) required_klds | required_kmods -- to have some official aliases
>        supported, but it's expected to bring confusion instead
> 
>     e) your variant
> 
> 
> Project B: automatic kldload
> ----------------------------
> 
> Having Project A implemented and the existing tests augmented with the
> respective metadata we could ask Kyua to automatically load the required
> kernel modules.
> 
>   From the implementation perspective, Kyua is expected to have a generic
> concept of a requirement resolver. A resolver may read all metadata and
> decide on its actions. If we decide on "klds" name in Project A then the
> very first resolver implemented would look for required_klds metadata, if
> it's present, then the mentioned kernel modules will be loaded. Such
> resolver could be named "klds", and we can think of future additions like
> "pkgs" resolver and so on.
> 
> Currently, I see it as a new kyua command to make it separate from the
> testing process itself. The interface could be as follows:
> 
> - List all available requirement resolvers (name and short description):
> 
>       > kyua rr list
> 
> - Run all resolvers or only specified ones. It would deal with the current
>     dir, and later we can think of "-k" option like "kyua test" command has.
> 
>       > kyua rr run [resolver1-name resolver2-name ...]
> 
> - It's proposed to be a sub-set of commands in case if in the future we want
>     additions, e.g. dry-run or verify command without actual resolving.
> 
> Having this interface the FreeBSD CI could have a separate step to run "kyua
> rr run" before the existing step which runs "kyua test". But from
> development perspective it may quickly end up with a need to have a single
> command for both. The following interface addition could be there:
> 
> - Run all or specified requirement resolvers before the actual testing:
> 
>       > kyua test --rr ["resolver1-name resolver2-name ..."]
> 
> Another option is to use the existing Kyua mechanisms and add it as a
> configuration variable, i.e. it can be configured once in the local
> kyua.conf not to specify it with every test command invocation:
> 
>       # without configuration:
>       > kyua -v rr="klds pkgs" test
> 
>       # with configuration:
>       > echo 'rr = "klds pkgs"' >> /etc/kyua/kyua.conf
>       > kyua test  # now it does resolving first
> 
> 
> Q2: Project B has the same set of questions about naming and style. For
> example, "rr" name for kyua.conf feels non-obvious and probably it could be
> named like "requirement_resolvers" to keep it self-explainable.
> 
> 
> What do you think? Any idea, comment, or suggestion are welcome.
> 
> 
> Best regards,
> igoro