igb/vlan problem after upgrading from 12.2 to 12.3

From: Guido van Rooij <guido_at_gvr.org>
Date: Thu, 23 Dec 2021 15:04:53 UTC
Hi,

I have an APU2 system with intel I210 nics. After upgrading from
12.2 to 12.3 I noticed problems. It seems like larger packets somehow
fail on igb0. This happens persistently, but when I bring the
interface down and up again, the problem goes away.

Here is the relavant config:
# WAN interface
ifconfig_igb0="mtu 1508 up -wol -arp"
vlans_igb0="vlan4 vlan6"
create_args_vlan4="vlan 4"
create_args_vlan6="vlan 6 -arp"
# Internet VLAN
ifconfig_igb0_ipv6="inet6 ifdisabled -nud -auto_linklocal -accept_rtadv"
ifconfig_vlan6_ipv6="inet6 ifdisabled -nud -auto_linklocal -accept_rtadv"

# IPTV VLAN
ifconfig_vlan4="SYNCDHCP"

Here is relevant part of the ifconfig output:
igb0: flags=8ac3<UP,BROADCAST,RUNNING,NOARP,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1508
	options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:0d:b9:4c:8b:e0
	inet6 fe80::20d:b9ff:fe4c:8be0%igb0 prefixlen 64 tentative scopeid 0x1
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=8<IFDISABLED>
vlan4: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1508
	options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:0d:b9:4c:8b:e0
	inet xx.xx.xxx.xx netmask 0xfffff800 broadcast xx.xx.xxx.255
	groups: vlan
	vlan: 4 vlanpcp: 0 parent interface: igb0
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=69<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL,NO_RADR>
vlan6: flags=88c3<UP,BROADCAST,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1508
	options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:0d:b9:4c:8b:e0
	groups: vlan
	vlan: 6 vlanpcp: 0 parent interface: igb0
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=48<IFDISABLED,NO_RADR>
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1500
	inet xx.xxx.xxx.xx --> xxx.xxx.x.xxx netmask 0xffffffff
	inet6 fe80::20d:b9ff:fe4c:8be0%ng0 prefixlen 64 scopeid 0xc
	nd6 options=22<ACCEPT_RTADV,AUTO_LINKLOCAL>

I do have two additional nics: igb2 is unused and igb1 has two vlan's
(vlan100 and vlan200) but the settings on that interface do not
have to be changed to get things working.

Note that on vlan6 we run PPPoE via mpd5. 
After the PPPoE link comes up, I can ssh into my colo box (on the
internet). A ping -s 1460 back does work, but ping -s 1480 does not.
After bringing down and up igb0, ping -s 1480 does work again.
(bringing down and up vlan6 does not let he ing come through)

My gut feeling tells me this has to do with the mtu being 1508 on
igb0. Perhaps this somehow does not get set correctly in the vlans
initially but does so when bringing the iterface down and up again.
Note again that this problem does not occur on 12.2.

Perhaps PR 260260 is related?

Hope you can help me debugging this further and get to a solution.

Thanks.

-Guido



-- 
Guido van Rooij	                      |  Phone: ++31 653 994 773
Gonz Beheer                           |
guido@gvr.org                         |  FreeBSD dinosaur