From nobody Wed Nov 06 18:12:10 2024 X-Original-To: freebsd-testing@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkCwG6Bwgz1GnqC; Wed, 06 Nov 2024 18:12:14 +0000 (UTC) (envelope-from igoro@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkCwG44Plz4lPK; Wed, 6 Nov 2024 18:12:14 +0000 (UTC) (envelope-from igoro@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730916734; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=yg9tOEi6bDxZIMhIj2tO1klUqZJHoiyUMV91qAE+uek=; b=FZK4QOBOIv6lgYcwjbvsFv5WjyAVPM5v6cz9YdYs1YdSr3I82LYXUirzitelkSIlnk5TE/ DzAzJg4Jfmv4kvA/W1LG6voAH2a5zrTX9Vd+4xUBfZ77mpEa4W2/skeN72B/WDn7+AOeTM lQVUoFJhp0Sz7Scm52ER0nnicCUQp5GY603aQPG3IwFOmbefkpSMpm44Dv9LESqHRYpeyR O7Bbbl/VDWl/rmyB+8JHXpThPa/8YnePG3AHbFp10rNukViseqvWPI33mND0aleoC+ppSl Zn+x1KuZMjaY8HQ4HMpEwM6ZaUPEqgsyy870mrBFiUIdUMjUMXIC7wkjBFZFmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730916734; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=yg9tOEi6bDxZIMhIj2tO1klUqZJHoiyUMV91qAE+uek=; b=w4awXEUpbWAk0oRMDGXRwL1CvK9N2mkm9iOt3xGT2uc3QNQgGV5m4jDX7uLKeYjCbAZv8D T6F64AuN0QYkYM6Zk8gMIe9ys5UY0UoEZZy7p96XZfF02G9Q9Y4EQ4aC8LDNHUfDw6GAVe UYL4zeCI2Fw0BqEpGhbD8e+Bw7COgbV6sIiGmkjXqaFxCF1FJHKv2cPbnnetIQ99xBumxT FgHBnC4jFRNMK8kYBR8xtqUeP75Q4AaMIkx82A/aXDcsYTdjHi347naGiAgkpImDtjKGYF zzbY9Engggb7NH6sxuf7xgAOSFiRqG1KYL86PEvLwicw3HJGCkUyXilP7v4qag== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730916734; a=rsa-sha256; cv=none; b=JPyZrdJXaoOPBoYMjgjV6aGcyHiTHlD/RN5PtnVIYae9sPcaL7JAw+gEIGFCZILsJYXsnV ZprYuglRyNnidqQYUs+vPzTpLjwLb2zvD1WissLyfQ5YPkBO0OI7dfzdbR1mKXqsAh7coo qLN+81ZJqjV2Bt9mU02XliEZX/Q69S8FjVjDA/vN5Vky9CfRDSa2ebbc8iTZBIjub7bHTv aNnF2HKNfD2j013E8sKwiE34F3G153zYbqHqoWxKFnoHdD7JVnTZlvfdy5caHbXI4RTQpA BPTaHx2kSU00yEBMYaC8OjY0/okbvMf0IajvmuMzPI3iLTsIUcVpB89nskmiQQ== Received: from [192.168.0.8] (unknown [78.83.216.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: igoro/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XkCwF4NnWzyyj; Wed, 6 Nov 2024 18:12:13 +0000 (UTC) (envelope-from igoro@FreeBSD.org) Message-ID: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org> Date: Wed, 6 Nov 2024 20:12:10 +0200 List-Id: Testing List-Archive: https://lists.freebsd.org/archives/freebsd-testing List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-testing@freebsd.org Sender: owner-freebsd-testing@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US From: Igor Ostapenko To: freebsd-testing@freebsd.org, freebsd-hackers@freebsd.org Subject: RFC: Add required_klds metadata to Kyua Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi FreeBSD developers, During Kyua's execenv=jail feature implementation there were discussions to get some help from Kyua side with kernel modules required by specific tests. The thing got "required_klds" codename. The problem to solve here is a well-known one: typically we invoke "kyua test" and get a lot of "Skipped" due to many kernel module requirements. And the pain is to know which ones need loading, especially if we target a sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new test is added with new module requirements. And other related issues. I would like to share the existing ideas and vision before the implementation itself. I propose to split it onto several subsequent projects to ease reasoning, review, etc. Project A: declaration & skipping --------------------------------- Add a new metadata property to Kyua which is used by a test to define its requirements on specific kernel modules expected being loaded. The following interface is expected: - Define it on ATF level, e.g. for atf-sh: atf_set require.klds pf pflog - Define it on Kyuafile level: test_program{name="testbinary1", required_klds="pf pflog"} - Get it listed per test case: > kyua list -v testbinary1 | grep required_klds required_klds = pf pflog if_epair Having it declared we could simplify tests and avoid manual checks for modules with the respective "skip test" lib calls. Thus, this new metadata would work the same way as the existing required_programs and required_files ones -- if some of the required modules are not loaded Kyua will skip such test with a respective message including names of the missing modules. An extra benefit is that we do not need to iterate it like: 1) kyua test, 2) kldload a-missing-module, 3) repeat. Such iteration is usually needed due to our tests typically do kldstat sequentially with "skip fast" logic. Skipping reason information still is not aggregated to make it easy to run a single kldload for all missing modules, but that's another story which is covered by the following Project B. Q1: I have doubts regarding the name of this new metadata. There are several other options in my mind, it would be appreciated to hear opinions regarding this: a) required_klds -- the original idea based on the vibe that FreeBSD Kyua fork is mainly for FreeBSD b) required_mods -- to keep it OS agnostic while Kyua still is treated as an OS agnostic tool, and freebsd/kyua GitHub repo actually is the main Kyua repo in the world today c) required_kmods -- a little improvement of the previous option to make sure it talks about "kernel modules" and leave space for other meanings in the future, just in case d) required_klds | required_kmods -- to have some official aliases supported, but it's expected to bring confusion instead e) your variant Project B: automatic kldload ---------------------------- Having Project A implemented and the existing tests augmented with the respective metadata we could ask Kyua to automatically load the required kernel modules. From the implementation perspective, Kyua is expected to have a generic concept of a requirement resolver. A resolver may read all metadata and decide on its actions. If we decide on "klds" name in Project A then the very first resolver implemented would look for required_klds metadata, if it's present, then the mentioned kernel modules will be loaded. Such resolver could be named "klds", and we can think of future additions like "pkgs" resolver and so on. Currently, I see it as a new kyua command to make it separate from the testing process itself. The interface could be as follows: - List all available requirement resolvers (name and short description): > kyua rr list - Run all resolvers or only specified ones. It would deal with the current dir, and later we can think of "-k" option like "kyua test" command has. > kyua rr run [resolver1-name resolver2-name ...] - It's proposed to be a sub-set of commands in case if in the future we want additions, e.g. dry-run or verify command without actual resolving. Having this interface the FreeBSD CI could have a separate step to run "kyua rr run" before the existing step which runs "kyua test". But from development perspective it may quickly end up with a need to have a single command for both. The following interface addition could be there: - Run all or specified requirement resolvers before the actual testing: > kyua test --rr ["resolver1-name resolver2-name ..."] Another option is to use the existing Kyua mechanisms and add it as a configuration variable, i.e. it can be configured once in the local kyua.conf not to specify it with every test command invocation: # without configuration: > kyua -v rr="klds pkgs" test # with configuration: > echo 'rr = "klds pkgs"' >> /etc/kyua/kyua.conf > kyua test # now it does resolving first Q2: Project B has the same set of questions about naming and style. For example, "rr" name for kyua.conf feels non-obvious and probably it could be named like "requirement_resolvers" to keep it self-explainable. What do you think? Any idea, comment, or suggestion are welcome. Best regards, igoro