Re: FreeBSD 12.3-p5: problems vnet on if_bridge

On Tue, 10 May 2022 21:21:29 +0200
FreeBSD User <freebsd@walstatt-de.de> wrote:

> Hello,
> 
> I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC
> and vnt jails attached to that second NIC (basically, the host is a recent Xigmanas with
> Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3).
> 
> The host is compromised of two NICs, em0 (management only) and igb0 (service/jails).
> Both, the server and the jails as well as the igb0 interface are residing on the same
> network, but both NICs are connected to two different ports on a switch, to which we do
> not have access (part of the campus infrastructure).
> 
> Both NICs are attached with a IPv4 of the same network, the host is listening on both
> NICs for services, i.e. port 22 for ssh. No problem to connect to both(!) addresses via
> ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet jails, each jail
> does have an if_epair created via "jib" and these vnet epairs are members of the bridge,
> to which ifb0 is also member.
> 
> Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is accessible
> flawlessly, accessing an jail is almost impossible. Pinging a jail does work after a
> while the ping initiating host has been waiting, in ery rare situations someone can
> access the sshd of the jail, but any access of that kind is highly erratic. From 5
> jails, at most two are responding to pings, the other don't and it is non-deterministic
> which host will respond. 
> 
> Following some advices found on the web, the following sysctl settings are provided to
> if_bridge: 
> 
> device	if_bridge
> net.link.bridge.ipfw: 0
> net.link.bridge.allow_llz_overlap: 0
> net.link.bridge.inherit_mac: 0
> net.link.bridge.log_stp: 0
> net.link.bridge.pfil_local_phys: 0
> net.link.bridge.pfil_member: 0
> net.link.bridge.ipfw_arp: 0
> net.link.bridge.pfil_bridge: 0
> net.link.bridge.pfil_onlyip: 0
> 
> We do not have access to the switch the box is connected to, so I don't have access to
> any logs revealing a problem either to a conceptual misunderstanding of networking of
> mine and so a misconfiguration or a probelm with Layer 2 or the switches themselfes.
> 
> I'd like to ask whether someone has a similar setup up and running and could report this
> - or give a hint of the problem I possibly made (igb0 is attached to an IPv4 AND is
> member of an if_brige on which IPv4 attached vnet jails are residing).
> 
> We have also already setup another "similar" scenarion with the same FreeBSD 12.3-p5
> version and also two NICs, but our "service/jail" NIC is part of a different IPv4
> network and the NIC is attached to a different switch (to which we have full access).
> 
> Thanks in advance,
> 
> O. Hartmann
> 

On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware chesum support, I
see a lot of :
[...]
Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq 101269476:101270000, ack 5077, win
257, options [nop,nop,TS val 2618589801 ecr 3610923914], length 524

Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> correct.

em0 is:

em0@pci0:0:25:0:	class=0x020000 card=0x20528086 chip=0x153b8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I217-V'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xf7d00000, size 131072, enabled
    bar   [14] = type Memory, range 32, base 0xf7d35000, size 4096, enabled
    bar   [18] = type I/O Port, range 32, base 0xf080, size 32, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 13[e0] = PCI Advanced Features: FLR TP


I remember faintly that there was an issue when I used to use FBSD 12