From nobody Wed Nov 06 20:27:05 2024 X-Original-To: freebsd-testing@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkGvw6LtQz5bnW1; Wed, 06 Nov 2024 20:27:08 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkGvw51F4z42TY; Wed, 6 Nov 2024 20:27:08 +0000 (UTC) (envelope-from kp@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730924828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U7kGfwXdGcXhyr5ABaJ65mkh2vAUg8G1YqyVz47zDlk=; b=Nh7q6WOdvohUXulSbOh/G7SF//QzzBd5fjPDospvJnWtUZ8jpGG6XmCYadSdQThgARvi24 wd0TnLP4V90mxyFQkWf+dbhqKG1XvUGwygun5vZK1Jg0+PeI5jZxXb4gGn9TUX2xhB8kal KpCYkpa4iheCQoG+BYdgDoLqxbTQ/ZXFbVHRJS17RRysdPjx+cS1jsHKTo+1/ThSgYUaat BhJWNu/lxFvcn62MZppT0tEDsT3UzaGaPbKgk57+qsAJWj6WAZeZRe/cjLVJyjlopRUtpF YihE4GtbDLM+99fxWO7N/UJ1/57rXqfbVX/kKOR7rt6LK6Vq7tmCErrYXanWPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730924828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U7kGfwXdGcXhyr5ABaJ65mkh2vAUg8G1YqyVz47zDlk=; b=dWfB7vsEHZtkC0SDa5aWdbWWoRzzBHh/cThcfiTtUJjQoSGQTpiWrSpm2Ywm2J1m/IG1bE qzBERAI8IZGRDCUxAcQZUZp2l/XR6dfA4tLiMyHCOHGSaw5z6v5e19TeNn5RH1T3Jxm/xA ec3C9rysbG2BSybmN+jZna+ibvgfcYDzQADTrt8EDr/Wu++2wSPBHmLJIPINg+0BnuEmGp 8K9S2pAvSkzM4JyyJyLKVHwmygEqHLCIPuVo4S4svHniTVe8fIUyuC5b8TqzkOO88ubafO 98aCq8J4xLjVztYwE2g4Q4NEvfbyZLmw0MIUT2XtPa2LObnxSgEYtF32DYVFbA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730924828; a=rsa-sha256; cv=none; b=CdRELeQ2R0u7BiuCHO6YQfOExzUn8Eu+JxFHbilJMxWd+6cbWBDKi1pL6SHDRRBmrlkaNf ajXoL6k9PlfqC/DdwVethCJUD/yLFfmn5qNzEsNYoLrwJTo0S2mjKjYpou6XqAYzlTRm/x 7fHOfRbXOWUQwA8aRxl138w6+tLgLmIx5jakEIr43LCdq7GMVejqA8ZZOfRhSAviBbzb35 A3FhWI00eWZVreiuTKWSE3jKRs8yxknGfREespcnfDkPv6q13wvp+wIA+Ud6VseQ5bRXgD UDRE+2SoVhj/OD9+uj09SfBK88BsVuClSNNm4fQvQYquIkluSG8gtTOzcx3K3g== Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.codepro.be", Issuer "R11" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XkGvw37ssz11nq; Wed, 6 Nov 2024 20:27:08 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id D98BA37923; Wed, 06 Nov 2024 21:27:05 +0100 (CET) From: Kristof Provost To: =?utf-8?q?Olivier_Cochard-Labb=C3=A9?= Cc: Igor Ostapenko , freebsd-testing@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: RFC: Add required_klds metadata to Kyua Date: Wed, 06 Nov 2024 21:27:05 +0100 X-Mailer: MailMate (1.14r5937) Message-ID: <9975A26D-6AB4-433F-B4B2-515F956BF366@FreeBSD.org> In-Reply-To: References: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org> List-Id: Testing List-Archive: https://lists.freebsd.org/archives/freebsd-testing List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-testing@freebsd.org Sender: owner-freebsd-testing@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_MailMate_C07615AC-1A59-4556-90E2-ECBD1E2270CC_=" Content-Transfer-Encoding: 8bit --=_MailMate_C07615AC-1A59-4556-90E2-ECBD1E2270CC_= Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 6 Nov 2024, at 21:13, Olivier Cochard-Labbé wrote: > On Wed, Nov 6, 2024 at 7:12 PM Igor Ostapenko > wrote: > >> >> >> The problem to solve here is a well-known one: typically we invoke >> "kyua >> test" and get a lot of "Skipped" due to many kernel module >> requirements. >> And >> the pain is to know which ones need loading, especially if we target >> a >> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a >> new >> test is added with new module requirements. And other related issues. >> >> >> Hello Igor, > > Thanks again for working on this topic > And there are even more challenging cases: what about tests that > assess the > behavior of different modules and need to unload them as well? > For example, sys/netipsec/tunnel/* or sys/opencrypto/, which involve > loading and unloading modules between tests." > That’s a good point. I ran into that a few days ago when I looked at using the new execenv=jail option to parallelise the netipsec tests, and discovered that that wasn’t straightforward. I wonder if an `forbidden_klds` or something is the answer there. At first to skip a test if the forbidden module is present, later on perhaps to enforce unloading of the module so the test can run. That may not be straightforward through, because module state is global, and if we only support loading required modules that’s easy (gather the list and load, or load when starting a test that requires a new module), but if we support unloading them it’ll affect test scheduling. (i.e. we can’t load modules while a `forbidden_klds` test is running, and can’t unload while a `require_klds` test is running). That said, perhaps we don’t need to worry about that just yet. It’s a minority of tests that forbid modules, so those can just keep doing what they’re doing already (i.e. run exclusively, do the checks in the test script itself). It shouldn’t stop us from making the majority of tests better. Best regards, Kristof --=_MailMate_C07615AC-1A59-4556-90E2-ECBD1E2270CC_= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On 6 Nov 2024, at 21:13, Olivier Cochard-Labb=C3=A9 wrote= :

On Wed, Nov 6, 2024 at 7:12=E2=80=AF= PM Igor Ostapenko <igoro@freebsd.org> wrote:

The problem to solve here is a well-known one: typically we invoke "k= yua
test" and get a lot of "Skipped" due to many kernel module requirements.
And
the pain is to know which ones need loading, especially if we target a
sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new=
test is added with new module requirements. And other related issues.

=

Hello Igor,

Thanks again for working on this topic
And there are even more challenging cases: what about tests that assess t= he
behavior of different modules and need to unload them as well?
For example, sys/netipsec/tunnel/* or sys/opencrypto/, which involve
loading and unloading modules between tests."


That=E2=80=99s a good point. I ran into that a few days a= go when I looked at using the new execenv=3Djail option to parallelise th= e netipsec tests, and discovered that that wasn=E2=80=99t straightforward= =2E

I wonder if an forbidden_klds or something is the answer ther= e. At first to skip a test if the forbidden module is present, later on p= erhaps to enforce unloading of the module so the test can run.

That may not be straightforward through, because module s= tate is global, and if we only support loading required modules that=E2=80= =99s easy (gather the list and load, or load when starting a test that re= quires a new module), but if we support unloading them it=E2=80=99ll affe= ct test scheduling. (i.e. we can=E2=80=99t load modules while a forbidden_klds test is running, and can=E2=80=99t unload while a require_klds test is runn= ing).

That said, perhaps we don=E2=80=99t need to worry about t= hat just yet. It=E2=80=99s a minority of tests that forbid modules, so th= ose can just keep doing what they=E2=80=99re doing already (i.e. run excl= usively, do the checks in the test script itself). It shouldn=E2=80=99t s= top us from making the majority of tests better.

Best regards,
Kristof

--=_MailMate_C07615AC-1A59-4556-90E2-ECBD1E2270CC_=--