vnet/if_bridge: ARP issues: connection failure

Hello,

the problem I'm about to report is not specific to CURRENT, I report this to CURRENT
because I'm list member. The problem occurs on FreeBSD 12.3-RELEASE-p5.

Setup: connecting to vnet jails bound to a dedicated NIC via if_bridge results in an
erratic behaviour (from my point of view). The box has two NICs, em0 which is dedicated
for managing the host and igb0 for services like NFS, SMB and jails (the host is de facto
a Xigmanas box). The NIC igb0 also has an IPv4 which is accessible without problem (sshd
is listening on em0 and igb0 and any service bound to igb0 and its IP itself works like a
charme, execept anything that is connected via if_bridge/vnet). Both, em0 and igb0, are
member of the same network and connected to the same switch (I guess, it's the campus
infrastructure, I have no access to that).

Phenomenon: trying to ping a jail results initially in a long term waiting and often in
"Host is down" - but then, sudenly, the jail is responding. Trying to connect to
port 22/tcp of any jail doesn't work in 90% of the cases, but randomly, a host (out of
five) does respond and the connection can be established. Terminating the connection and
tryin again is in 99% then a fail. Once connected the ssh connection fries after a couple
of seconds of inactivity.

Checking ARP on the jail (login via host and jexec) and listening via
tcpdump -XXvi vnet0 arp
on a jail while pinging the jail from the netowrk shows up the typical requests, but not
every request is then acompanied by a reply. I'm not firm in terms of networking and
investigating ARP issues, so I followed some instructions found with ARP issues on FBSD,
vnet and routing.

MIB settings (on the host itself, vnet untouched):
net.inet.ip.forwarding: 0
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0

I also realised that on the igb0 interace checksum errors occured while rxcsum is
enabled. I disbaled special features via "ifconfig igb0  -rxcsum -txcsum -tso -lro

I'm out of ideas here.

Another box, the same base OS, similar setup (two NICs, same ambition), but with the
difference that the second NIC resides on a different network and is connected to a
different switch, also if_bridge and vnet attached jails, there is no problem.

Either there is a serious bug in 12.3-p5 (haven't had the chance to check on 13/CURRENT)
or I'm doing something terribly wrong.

Some technical details:




em0@pci0:0:25:0:        class=0x020000 card=0x29828016 chip=0x153b8086 rev=0x04 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I217-V'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xf7d00000, size 131072, enabled
    bar   [14] = type Memory, range 32, base 0xf7d35000, size 4096, enabled
    bar   [18] = type I/O Port, range 32, base 0xf080, size 32, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 13[e0] = PCI Advanced Features: FLR TP


gb0@pci0:4:0:0:        class=0x020000 card=0x00028086 chip=0x15848086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xf7900000, size 1048576, enabled
    bar   [1c] = type Memory, range 32, base 0xf7a00000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks 
    cap 11[70] = MSI-X supports 5 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR NS
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 xxxxxxxxxxxxxxxxxxxxx
    ecap 0017[1a0] = TPH Requester 1

Kind regards,

oh