Re: git: b60600ceeb68 - main - pf tests: Serialize

On 31 Mar 2023, at 9:05, Mark Johnston wrote:
> On Fri, Mar 31, 2023 at 08:56:56AM +0900, Kristof Provost wrote:
>> On 31 Mar 2023, at 8:36, Mark Johnston wrote:
>>> The branch main has been updated by markj:
>>>
>>> URL: https://cgit.FreeBSD.org/src/commit/?id=b60600ceeb68d1001d61222830e0be3441ef0885
>>>
>>> commit b60600ceeb68d1001d61222830e0be3441ef0885
>>> Author:     Mark Johnston <markj@FreeBSD.org>
>>> AuthorDate: 2023-03-25 12:55:41 +0000
>>> Commit:     Mark Johnston <markj@FreeBSD.org>
>>> CommitDate: 2023-03-30 23:35:59 +0000
>>>
>>>     pf tests: Serialize
>>>
>>>     These tests reuse jail names and cannot run in parallel.  Until this is
>>>     fixed - which is desirable since these takes take a while to run - tell
>>>     kyua to serialize them.
>>>
>>
>> The tests also recycle IP ranges, so merely changing the jail names is insufficient.
>
> Could you please give an example?  I looked at some of the tests but can
> only see cases where addresses are assigned within the vnet jail(s)
> created by the tests, in which case I'd expect no problems.
>
Altq:hfsq assigns 192.0.2.1/24 on an epair on the host, so does altq:match, altq:cbq_vlan, altq:codel_bridge, dup:dup_to, ether:mac, ether:proto and .. then I got bored looking.

It’s not just the pf tests either. For example the netinet/forward:fwd_ip_icmp_iface_fast_success test does that too (well, 192.0.2.1/29, but anyway). There are going to be more, this is just from a very quick look.

>> Realistically the easiest way to get these to run in parallel would be to run each test in its own vnet so both overlapping IP ranges and name conflicts don’t matter.
>
> Yeah, I was wondering whether it'd be possible to have kyua handle the
> creation and teardown of a per-test jail, if only to avoid having to go
> through all of the tests which use static jail names.

That’s what I was thinking as well, yes.

We might be able to get to a point where all the tests have unique jail names, but we’re never going to manage to de-conflict all of the IP assignments. Even if we could, it’d make writing tests harder and that’s counterproductive.

The only issue I can think of with having the tests run in their own vnet is that there may be some tests that play with non-vneted sysctls, so we will have to make that vnet feature optional.

Best regards,
Kristof