From nobody Sun Dec 22 16:08:27 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YGR0Z4vzDz5YwPf; Sun, 22 Dec 2024 16:08:46 +0000 (UTC) (envelope-from igoro@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YGR0Z2sBlz4Z8k; Sun, 22 Dec 2024 16:08:46 +0000 (UTC) (envelope-from igoro@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1734883726; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nk8KCtozKyxT1acNcx8Vfg+RZrZzkZIJUvqKODgLub0=; b=RHPaCoP1fhggMBx4B4qo2a5l9nhf0Slpqsau/FLFAWuzsr7NRWmKjRPOHaF9Qld7K+GoQc VlbVSOfVVBrcUMKlC/iBUHNhWSoF62X6WezWQJvRUFvuuVCuwCfv9oIInHHCsRtgWxP8VT WVECMeRGYiWhq5uWTkJofYWyd0z2UqsEMlkm8fV4FZlkNgXLQ8b5YW1pB1GK6Xfbw/5KPe KKG4RELO+Z+eOzmxFhPYXxPLztv4lE0IlgHuGchs7O3tibsLtTuAGfx2spJWaNxSUcPsDX 7pG8YDDZuiXvq6s3KEk3IUtzz9glRqGF2ZTiX+Fdeio9VCz+sNENi/on9Z5MHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1734883726; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nk8KCtozKyxT1acNcx8Vfg+RZrZzkZIJUvqKODgLub0=; b=aGFry0+LT0iaNNMd2j9Mye727I7McIVUn+btwBeVd2UIX8UB6r01eKdPiopQndCmuNG/zn vZvnZPPBYDLsg8jUASQq/U+I/2zRPq4OE0T+v4uJvzbZM39dIb9zE6kAyl4vTtXe4OQY0F v9UlAl2Pv/ZBXHnbL5EfsnGTfizuRxySElyZLNhnRqi797mkk77wjYgSrFEMCr3NbpLDm/ 46mwY8av+qTDeCc4lzarnz7utN65WhADh/0/ju3oUFMbNjH1VW/WP4t0X/Gy8Mve3WfLgn rG0uxOGRJRQjkZIUPPI+gMqvPt77om5g+qc4fHlF5pURulE2mNcrUd3D8FvLTw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1734883726; a=rsa-sha256; cv=none; b=icQMlkUjExpADFEI+bOB4l78GBiWRo07A/UEgNsUgQM1zARQqhLKtpNi/WNARhAg0gQi9O X5gx1vvl3RPPv25gV9mZgicMOZSSFJp+8pT2h4fLIzEoOGK9oSkprUaGUa3+rqL+UGiakX 3FyTJvQDv/Jr7qCq0h4dsT1SGoPtfY197ESFEUvM42oEwzRxz8mntZa8k0IO2gW5aB8jME ohDYtAMkqg/CvInREN9b3dGh25luTTv5dc3NQZc5jXojxUrymbzK8y3dOVvbZxObNAlR+A tUtIrbUPYJu1lDTlycAWWE1Ake7AilG9YwkdG0xNjUpAjORash8p5gwHtu2SIg== Received: from [192.168.0.8] (unknown [78.83.216.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: igoro/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4YGR0Y6fxXz1MGl; Sun, 22 Dec 2024 16:08:45 +0000 (UTC) (envelope-from igoro@FreeBSD.org) Message-ID: <67e3f4b3-ef83-4f65-88e4-e1228caee830@FreeBSD.org> Date: Sun, 22 Dec 2024 18:08:27 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Igor Ostapenko Subject: Re: RFC: Add required_klds metadata to Kyua To: freebsd-testing@freebsd.org, freebsd-hackers@freebsd.org References: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org> Content-Language: en-US In-Reply-To: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi, The Project B patch is open for testing and discussion: https://reviews.freebsd.org/D48087 The outline of the currently proposed UI/UX is as follows: # kyua prepare all Run all requirement resolvers kmods Resolve required_kmods for FreeBSD # grep -r kmods * Kyuafile:atf_test_program{name="divert-to", ... required_kmods="if_epair if_bridge pf ipdivert"} mbuf: atf_set require.kmods if_epair if_bridge pf dummymbuf # kyua prepare --dry-run all kldload dummymbuf if_bridge if_epair ipdivert pf # kyua prepare kmods kldload dummymbuf if_bridge if_epair ipdivert pf Testing and sharing your thoughts are welcome. Best regards, igoro Igor Ostapenko wrote on 11/6/24 8:12 PM: > Hi FreeBSD developers, > > During Kyua's execenv=jail feature implementation there were discussions to > get some help from Kyua side with kernel modules required by specific tests. > The thing got "required_klds" codename. > > The problem to solve here is a well-known one: typically we invoke "kyua > test" and get a lot of "Skipped" due to many kernel module requirements. And > the pain is to know which ones need loading, especially if we target a > sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new > test is added with new module requirements. And other related issues. > > I would like to share the existing ideas and vision before the > implementation itself. I propose to split it onto several subsequent > projects to ease reasoning, review, etc. > > > Project A: declaration & skipping > --------------------------------- > > Add a new metadata property to Kyua which is used by a test to define its > requirements on specific kernel modules expected being loaded. The following > interface is expected: > > - Define it on ATF level, e.g. for atf-sh: > > atf_set require.klds pf pflog > > - Define it on Kyuafile level: > > test_program{name="testbinary1", required_klds="pf pflog"} > > - Get it listed per test case: > > > kyua list -v testbinary1 | grep required_klds > required_klds = pf pflog if_epair > > Having it declared we could simplify tests and avoid manual checks for > modules with the respective "skip test" lib calls. Thus, this new metadata > would work the same way as the existing required_programs and required_files > ones -- if some of the required modules are not loaded Kyua will skip such > test with a respective message including names of the missing modules. > > An extra benefit is that we do not need to iterate it like: 1) kyua test, 2) > kldload a-missing-module, 3) repeat. Such iteration is usually needed due to > our tests typically do kldstat sequentially with "skip fast" logic. Skipping > reason information still is not aggregated to make it easy to run a single > kldload for all missing modules, but that's another story which is covered > by the following Project B. > > Q1: I have doubts regarding the name of this new metadata. There are several > other options in my mind, it would be appreciated to hear opinions regarding > this: > a) required_klds -- the original idea based on the vibe that FreeBSD Kyua > fork is mainly for FreeBSD > > b) required_mods -- to keep it OS agnostic while Kyua still is treated > as an OS agnostic tool, and freebsd/kyua GitHub repo actually is the > main Kyua repo in the world today > > c) required_kmods -- a little improvement of the previous option to make > sure it talks about "kernel modules" and leave space for other meanings > in the future, just in case > > d) required_klds | required_kmods -- to have some official aliases > supported, but it's expected to bring confusion instead > > e) your variant > > > Project B: automatic kldload > ---------------------------- > > Having Project A implemented and the existing tests augmented with the > respective metadata we could ask Kyua to automatically load the required > kernel modules. > > From the implementation perspective, Kyua is expected to have a generic > concept of a requirement resolver. A resolver may read all metadata and > decide on its actions. If we decide on "klds" name in Project A then the > very first resolver implemented would look for required_klds metadata, if > it's present, then the mentioned kernel modules will be loaded. Such > resolver could be named "klds", and we can think of future additions like > "pkgs" resolver and so on. > > Currently, I see it as a new kyua command to make it separate from the > testing process itself. The interface could be as follows: > > - List all available requirement resolvers (name and short description): > > > kyua rr list > > - Run all resolvers or only specified ones. It would deal with the current > dir, and later we can think of "-k" option like "kyua test" command has. > > > kyua rr run [resolver1-name resolver2-name ...] > > - It's proposed to be a sub-set of commands in case if in the future we want > additions, e.g. dry-run or verify command without actual resolving. > > Having this interface the FreeBSD CI could have a separate step to run "kyua > rr run" before the existing step which runs "kyua test". But from > development perspective it may quickly end up with a need to have a single > command for both. The following interface addition could be there: > > - Run all or specified requirement resolvers before the actual testing: > > > kyua test --rr ["resolver1-name resolver2-name ..."] > > Another option is to use the existing Kyua mechanisms and add it as a > configuration variable, i.e. it can be configured once in the local > kyua.conf not to specify it with every test command invocation: > > # without configuration: > > kyua -v rr="klds pkgs" test > > # with configuration: > > echo 'rr = "klds pkgs"' >> /etc/kyua/kyua.conf > > kyua test # now it does resolving first > > > Q2: Project B has the same set of questions about naming and style. For > example, "rr" name for kyua.conf feels non-obvious and probably it could be > named like "requirement_resolvers" to keep it self-explainable. > > > What do you think? Any idea, comment, or suggestion are welcome. > > > Best regards, > igoro