Re: FreeBSD 12.3-p5: problems vnet on if_bridge

Hello,

could you solve the problem? I think I ran into the same problem.
I opened a Ticket.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264198

I seems to be related to IPFW and effects vnet-Jails and also bhyve VMs.

regards
Ole


Wed, 11 May 2022 20:47:55 +0200 - FreeBSD User <freebsd@walstatt-de.de>:

> On Tue, 10 May 2022 21:21:29 +0200
> FreeBSD User <freebsd@walstatt-de.de> wrote:
> 
> > Hello,
> > 
> > I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5
> > host having a second NIC and vnt jails attached to that second NIC
> > (basically, the host is a recent Xigmanas with Bastille jails, but
> > the issue also occurs on a vanilla FreeBSD 12.3).
> > 
> > The host is compromised of two NICs, em0 (management only) and igb0
> > (service/jails). Both, the server and the jails as well as the igb0
> > interface are residing on the same network, but both NICs are
> > connected to two different ports on a switch, to which we do not
> > have access (part of the campus infrastructure).
> > 
> > Both NICs are attached with a IPv4 of the same network, the host is
> > listening on both NICs for services, i.e. port 22 for ssh. No
> > problem to connect to both(!) addresses via ssh. igb0 is member of
> > an if_bridge. The box also hosts a bunch of vnet jails, each jail
> > does have an if_epair created via "jib" and these vnet epairs are
> > members of the bridge, to which ifb0 is also member.
> > 
> > Problem: while any service bound to NIC igb0/IPv4 residing on igb0
> > is accessible flawlessly, accessing an jail is almost impossible.
> > Pinging a jail does work after a while the ping initiating host has
> > been waiting, in ery rare situations someone can access the sshd of
> > the jail, but any access of that kind is highly erratic. From 5
> > jails, at most two are responding to pings, the other don't and it
> > is non-deterministic which host will respond. 
> > 
> > Following some advices found on the web, the following sysctl
> > settings are provided to if_bridge: 
> > 
> > device	if_bridge
> > net.link.bridge.ipfw: 0
> > net.link.bridge.allow_llz_overlap: 0
> > net.link.bridge.inherit_mac: 0
> > net.link.bridge.log_stp: 0
> > net.link.bridge.pfil_local_phys: 0
> > net.link.bridge.pfil_member: 0
> > net.link.bridge.ipfw_arp: 0
> > net.link.bridge.pfil_bridge: 0
> > net.link.bridge.pfil_onlyip: 0
> > 
> > We do not have access to the switch the box is connected to, so I
> > don't have access to any logs revealing a problem either to a
> > conceptual misunderstanding of networking of mine and so a
> > misconfiguration or a probelm with Layer 2 or the switches
> > themselfes.
> > 
> > I'd like to ask whether someone has a similar setup up and running
> > and could report this
> > - or give a hint of the problem I possibly made (igb0 is attached
> > to an IPv4 AND is member of an if_brige on which IPv4 attached vnet
> > jails are residing).
> > 
> > We have also already setup another "similar" scenarion with the
> > same FreeBSD 12.3-p5 version and also two NICs, but our
> > "service/jail" NIC is part of a different IPv4 network and the NIC
> > is attached to a different switch (to which we have full access).
> > 
> > Thanks in advance,
> > 
> > O. Hartmann
> > 
> 
> On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware
> chesum support, I see a lot of :
> [...]
> Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq
> 101269476:101270000, ack 5077, win 257, options [nop,nop,TS val
> 2618589801 ecr 3610923914], length 524
> 
> Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect ->
> correct.
> 
> em0 is:
> 
> em0@pci0:0:25:0:	class=0x020000 card=0x20528086
> chip=0x153b8086 rev=0x04 hdr=0x00 vendor     = 'Intel Corporation'
>     device     = 'Ethernet Connection I217-V'
>     class      = network
>     subclass   = ethernet
>     bar   [10] = type Memory, range 32, base 0xf7d00000, size 131072,
> enabled bar   [14] = type Memory, range 32, base 0xf7d35000, size
> 4096, enabled bar   [18] = type I/O Port, range 32, base 0xf080, size
> 32, enabled cap 01[c8] = powerspec 2  supports D0 D3  current D0
>     cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
>     cap 13[e0] = PCI Advanced Features: FLR TP
> 
> 
> I remember faintly that there was an issue when I used to use FBSD 12
> 
> 
> 
>