FreeBSD 12.3-p5: problems vnet on if_bridge

From: FreeBSD User <freebsd_at_walstatt-de.de>
Date: Tue, 10 May 2022 19:21:29 UTC
Hello,

I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC
and vnt jails attached to that second NIC (basically, the host is a recent Xigmanas with
Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3).

The host is compromised of two NICs, em0 (management only) and igb0 (service/jails).
Both, the server and the jails as well as the igb0 interface are residing on the same
network, but both NICs are connected to two different ports on a switch, to which we do
not have access (part of the campus infrastructure).

Both NICs are attached with a IPv4 of the same network, the host is listening on both
NICs for services, i.e. port 22 for ssh. No problem to connect to both(!) addresses via
ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet jails, each jail
does have an if_epair created via "jib" and these vnet epairs are members of the bridge,
to which ifb0 is also member.

Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is accessible
flawlessly, accessing an jail is almost impossible. Pinging a jail does work after a
while the ping initiating host has been waiting, in ery rare situations someone can
access the sshd of the jail, but any access of that kind is highly erratic. From 5 jails,
at most two are responding to pings, the other don't and it is non-deterministic which
host will respond. 

Following some advices found on the web, the following sysctl settings are provided to
if_bridge: 

device	if_bridge
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0

We do not have access to the switch the box is connected to, so I don't have access to
any logs revealing a problem either to a conceptual misunderstanding of networking of
mine and so a misconfiguration or a probelm with Layer 2 or the switches themselfes.

I'd like to ask whether someone has a similar setup up and running and could report this
- or give a hint of the problem I possibly made (igb0 is attached to an IPv4 AND is
member of an if_brige on which IPv4 attached vnet jails are residing).

We have also already setup another "similar" scenarion with the same FreeBSD 12.3-p5
version and also two NICs, but our "service/jail" NIC is part of a different IPv4 network
and the NIC is attached to a different switch (to which we have full access).

Thanks in advance,

O. Hartmann