[Bug 271205] [ix] [carp]: Continuous input errors on Intel X553
Date: Tue, 02 May 2023 15:42:59 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271205
Bug ID: 271205
Summary: [ix] [carp]: Continuous input errors on Intel X553
Product: Base System
Version: 12.4-RELEASE
Hardware: Any
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: bugs@FreeBSD.org
Reporter: duerrd561@gmail.com
I have a pair of FreeBSD 12.4 machines acting as a redundant gateway router for
a private network (fw1 and fw2). This is your classic pf + carp configuration
with a crossover cable between the pfsync interface in order to create a
active/backup firewall configuration. It has worked very well and the
configuration has been stable since the FreeBSD 10.x days, and up through
12.2-STABLE. I upgraded the systems to 12.4-RELEASE approximately 4 months ago,
and since then I am seeing continuous input errors accruing on the ix1 (WAN)
interfaces of both machines. The errors do not appear to have a material impact
on network performance, at least not that I know of, but they are detected by
my Check MK monitoring system and causing alert fatigue due to constant state
changes throughout the day.
The hardware is a pair of 1U Supermicro servers which each contain 4 Intel X553
interfaces in them:
ix0: <Intel(R) X553 (1GbE)> mem 0xddc00000-0xdddfffff,0xdde04000-0xdde07fff at
device 0.0 on pci6
ix1: <Intel(R) X553 (1GbE)> mem 0xdda00000-0xddbfffff,0xdde00000-0xdde03fff at
device 0.1 on pci6
ix2: <Intel(R) X553 (1GbE)> mem 0xdd600000-0xdd7fffff,0xdd804000-0xdd807fff at
device 0.0 on pci7
ix3: <Intel(R) X553 (1GbE)> mem 0xdd400000-0xdd5fffff,0xdd800000-0xdd803fff at
device 0.1 on pci7
The machines are kept up to date with `freebsd-update`, but have a basic custom
kernel config which is GENERIC with all unused hardware drivers disabled, and
with pf compiled in. Here's the `freebsd-version` output from fw1 and fw2
respectively:
[dd@fw1 dd]$ freebsd-version -kru
12.4-RELEASE-p2
12.4-RELEASE-p2
12.4-RELEASE-p2
[dd@fw2 dd]$ freebsd-version -kru
12.4-RELEASE-p2
12.4-RELEASE-p2
12.4-RELEASE-p2
Here's the relevant interface config from /etc/rc.conf for fw1 and fw2
respectively:
ifconfig_ix1="up media 1000baseTX mediaopt full-duplex -rxcsum -rxcsum6 -txcsum
-txcsum6 -lro -tso -vlanhwtso -vlanhwtag -vlanhwfilter -vlanhwcsum description
FW1-EXT"
ipv4_addrs_ix1="x.x.x.164/27"
ifconfig_ix1_alias0="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.166/32"
ifconfig_ix1_alias1="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.167/32"
ifconfig_ix1_alias2="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.168/32"
ifconfig_ix1_alias3="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.169/32"
ifconfig_ix1_alias4="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.170/32"
ifconfig_ix1_alias5="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.171/32"
ifconfig_ix1_alias6="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.172/32"
ifconfig_ix1_alias7="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.173/32"
ifconfig_ix1_alias8="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.174/32"
ifconfig_ix1_alias9="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.175/32"
ifconfig_ix1_alias10="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.176/32"
ifconfig_ix1_alias11="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.177/32"
ifconfig_ix1_alias12="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.178/32"
ifconfig_ix1_alias13="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.179/32"
ifconfig_ix1_alias14="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.180/32"
ifconfig_ix1_alias15="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.181/32"
ifconfig_ix1_alias16="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.182/32"
ifconfig_ix1_alias17="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.183/32"
ifconfig_ix1_alias18="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.184/32"
ifconfig_ix1_alias19="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.185/32"
ifconfig_ix1_alias20="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.186/32"
ifconfig_ix1_alias21="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.187/32"
ifconfig_ix1_alias22="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.188/32"
ifconfig_ix1_alias23="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.189/32"
ifconfig_ix1_alias24="inet vhid 2 advskew 10 pass XXXXX alias x.x.x.190/32"
ifconfig_ix1="up media 1000baseTX mediaopt full-duplex -rxcsum -rxcsum6 -txcsum
-txcsum6 -lro -tso -vlanhwtso -vlanhwtag -vlanhwfilter -vlanhwcsum description
FW2-EXT"
ipv4_addrs_ix1="x.x.x.165/27"
ifconfig_ix1_alias0="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.166/32"
ifconfig_ix1_alias1="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.167/32"
ifconfig_ix1_alias2="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.168/32"
ifconfig_ix1_alias3="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.169/32"
ifconfig_ix1_alias4="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.170/32"
ifconfig_ix1_alias5="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.171/32"
ifconfig_ix1_alias6="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.172/32"
ifconfig_ix1_alias7="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.173/32"
ifconfig_ix1_alias8="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.174/32"
ifconfig_ix1_alias9="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.175/32"
ifconfig_ix1_alias10="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.176/32"
ifconfig_ix1_alias11="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.177/32"
ifconfig_ix1_alias12="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.178/32"
ifconfig_ix1_alias13="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.179/32"
ifconfig_ix1_alias14="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.180/32"
ifconfig_ix1_alias15="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.181/32"
ifconfig_ix1_alias16="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.182/32"
ifconfig_ix1_alias17="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.183/32"
ifconfig_ix1_alias18="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.184/32"
ifconfig_ix1_alias19="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.185/32"
ifconfig_ix1_alias20="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.186/32"
ifconfig_ix1_alias21="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.187/32"
ifconfig_ix1_alias22="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.188/32"
ifconfig_ix1_alias23="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.189/32"
ifconfig_ix1_alias24="inet vhid 2 advskew 20 pass XXXXX alias x.x.x.190/32"
Here's the relevant ifconfig output from fw1 and fw2 respectively:
ix1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu
1500
description: FW1-EXT
options=803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
ether ac:1f:6b:48:75:bd
inet x.x.x.166 netmask 0xffffffff broadcast x.x.x.166 vhid 2
inet x.x.x.167 netmask 0xffffffff broadcast x.x.x.167 vhid 2
inet x.x.x.168 netmask 0xffffffff broadcast x.x.x.168 vhid 2
inet x.x.x.169 netmask 0xffffffff broadcast x.x.x.169 vhid 2
inet x.x.x.170 netmask 0xffffffff broadcast x.x.x.170 vhid 2
inet x.x.x.171 netmask 0xffffffff broadcast x.x.x.171 vhid 2
inet x.x.x.172 netmask 0xffffffff broadcast x.x.x.172 vhid 2
inet x.x.x.173 netmask 0xffffffff broadcast x.x.x.173 vhid 2
inet x.x.x.174 netmask 0xffffffff broadcast x.x.x.174 vhid 2
inet x.x.x.175 netmask 0xffffffff broadcast x.x.x.175 vhid 2
inet x.x.x.176 netmask 0xffffffff broadcast x.x.x.176 vhid 2
inet x.x.x.177 netmask 0xffffffff broadcast x.x.x.177 vhid 2
inet x.x.x.178 netmask 0xffffffff broadcast x.x.x.178 vhid 2
inet x.x.x.179 netmask 0xffffffff broadcast x.x.x.179 vhid 2
inet x.x.x.180 netmask 0xffffffff broadcast x.x.x.180 vhid 2
inet x.x.x.181 netmask 0xffffffff broadcast x.x.x.181 vhid 2
inet x.x.x.182 netmask 0xffffffff broadcast x.x.x.182 vhid 2
inet x.x.x.183 netmask 0xffffffff broadcast x.x.x.183 vhid 2
inet x.x.x.184 netmask 0xffffffff broadcast x.x.x.184 vhid 2
inet x.x.x.185 netmask 0xffffffff broadcast x.x.x.185 vhid 2
inet x.x.x.186 netmask 0xffffffff broadcast x.x.x.186 vhid 2
inet x.x.x.187 netmask 0xffffffff broadcast x.x.x.187 vhid 2
inet x.x.x.188 netmask 0xffffffff broadcast x.x.x.188 vhid 2
inet x.x.x.189 netmask 0xffffffff broadcast x.x.x.189 vhid 2
inet x.x.x.190 netmask 0xffffffff broadcast x.x.x.190 vhid 2
inet x.x.x.164 netmask 0xffffffe0 broadcast x.x.x.191
carp: MASTER vhid 2 advbase 1 advskew 10
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu
1500
description: FW2-EXT
options=803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
ether ac:1f:6b:48:77:d5
inet x.x.x.166 netmask 0xffffffff broadcast x.x.x.166 vhid 2
inet x.x.x.167 netmask 0xffffffff broadcast x.x.x.167 vhid 2
inet x.x.x.168 netmask 0xffffffff broadcast x.x.x.168 vhid 2
inet x.x.x.169 netmask 0xffffffff broadcast x.x.x.169 vhid 2
inet x.x.x.170 netmask 0xffffffff broadcast x.x.x.170 vhid 2
inet x.x.x.171 netmask 0xffffffff broadcast x.x.x.171 vhid 2
inet x.x.x.172 netmask 0xffffffff broadcast x.x.x.172 vhid 2
inet x.x.x.173 netmask 0xffffffff broadcast x.x.x.173 vhid 2
inet x.x.x.174 netmask 0xffffffff broadcast x.x.x.174 vhid 2
inet x.x.x.175 netmask 0xffffffff broadcast x.x.x.175 vhid 2
inet x.x.x.176 netmask 0xffffffff broadcast x.x.x.176 vhid 2
inet x.x.x.177 netmask 0xffffffff broadcast x.x.x.177 vhid 2
inet x.x.x.178 netmask 0xffffffff broadcast x.x.x.178 vhid 2
inet x.x.x.179 netmask 0xffffffff broadcast x.x.x.179 vhid 2
inet x.x.x.180 netmask 0xffffffff broadcast x.x.x.180 vhid 2
inet x.x.x.181 netmask 0xffffffff broadcast x.x.x.181 vhid 2
inet x.x.x.182 netmask 0xffffffff broadcast x.x.x.182 vhid 2
inet x.x.x.183 netmask 0xffffffff broadcast x.x.x.183 vhid 2
inet x.x.x.184 netmask 0xffffffff broadcast x.x.x.184 vhid 2
inet x.x.x.185 netmask 0xffffffff broadcast x.x.x.185 vhid 2
inet x.x.x.186 netmask 0xffffffff broadcast x.x.x.186 vhid 2
inet x.x.x.187 netmask 0xffffffff broadcast x.x.x.187 vhid 2
inet x.x.x.188 netmask 0xffffffff broadcast x.x.x.188 vhid 2
inet x.x.x.189 netmask 0xffffffff broadcast x.x.x.189 vhid 2
inet x.x.x.190 netmask 0xffffffff broadcast x.x.x.190 vhid 2
inet x.x.x.165 netmask 0xffffffe0 broadcast x.x.x.191
carp: BACKUP vhid 2 advbase 1 advskew 20
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Here's the netstat command and output that is being scraped by Check MK's
monitoring agent, showing the accrued 'Ierrs' since last reboot, for fw1 and
fw2 respectively:
[dd@fw1 dd]$ netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link
ix0 1500 <Link#1> ac:1f:6b:48:75:bc 562360 0 0 143914220
1275732 0 731183852 0
ix1 1500 <Link#2> ac:1f:6b:48:75:bd 16542924 9844 0 5580697236
13399023 0 6278712421 0
ix2 1500 <Link#3> ac:1f:6b:48:75:be 7838210 0 0 3723784709
7797874 0 2672566967 0
ix3 1500 <Link#4> ac:1f:6b:48:75:be 5858929 0 0 2510511102
8084444 0 2889334431 0
enc0 1536 <Link#6> enc0 999484 0 0 199309354
1791492 0 1818813983 0
pfsync0 1500 <Link#8> pfsync0 561059 0 0 133067492
1275449 0 708197332 0
lagg0 1500 <Link#9> ac:1f:6b:48:75:be 13697139 0 0 6234295811
15882318 10 5561901398 0
lagg0.6 1500 <Link#10> ac:1f:6b:48:75:be 1346494 0 0 1224788631
895434 4 425847084 0
lagg0.7 1500 <Link#11> ac:1f:6b:48:75:be 11312155 0 0 3937770163
14185843 3 4927307623 0
lagg0.10 1500 <Link#12> ac:1f:6b:48:75:be 803446 0 0 934196358
613978 3 57663690 0
[dd@fw2 dd]$ netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link
ix0 1500 <Link#1> ac:1f:6b:48:77:d4 1283178 0 0 735226863
567969 0 146696669 0
ix1 1500 <Link#2> ac:1f:6b:48:77:d5 1213797 1206 0 109701049
269221 0 56238161 0
ix2 1500 <Link#3> ac:1f:6b:48:77:d6 523113 0 0 60681990
283135 0 45325911 0
ix3 1500 <Link#4> ac:1f:6b:48:77:d6 557343 0 0 54758663
277200 0 44831157 0
enc0 1536 <Link#6> enc0 11526 0 0 1236171
16305 0 7547132 0
pfsync0 1500 <Link#8> pfsync0 1281713 0 0 711380284
567848 0 136556706 0
lagg0 1500 <Link#9> ac:1f:6b:48:77:d6 1080456 0 0 115440653
560335 9 90157068 0
lagg0.6 1500 <Link#10> ac:1f:6b:48:77:d6 160122 0 0 19010320
59811 3 13390952 0
lagg0.7 1500 <Link#11> ac:1f:6b:48:77:d6 589109 0 0 51511565
310604 3 49364530 0
lagg0.10 1500 <Link#12> ac:1f:6b:48:77:d6 94802 0 0 8141900
4320 3 473324 0
Here's the output of `sysctl dev.ix.1.mac_stats` from fw1 amd fw2 respectively:
[dd@fw1 dd]$ sysctl dev.ix.1.mac_stats
dev.ix.1.mac_stats.tx_frames_1024_1522: 2731033
dev.ix.1.mac_stats.tx_frames_512_1023: 1069308
dev.ix.1.mac_stats.tx_frames_256_511: 527405
dev.ix.1.mac_stats.tx_frames_128_255: 5138744
dev.ix.1.mac_stats.tx_frames_65_127: 3667443
dev.ix.1.mac_stats.tx_frames_64: 345704
dev.ix.1.mac_stats.management_pkts_txd: 0
dev.ix.1.mac_stats.mcast_pkts_txd: 90006
dev.ix.1.mac_stats.bcast_pkts_txd: 67
dev.ix.1.mac_stats.good_pkts_txd: 13479637
dev.ix.1.mac_stats.total_pkts_txd: 13479637
dev.ix.1.mac_stats.good_octets_txd: 6322986085
dev.ix.1.mac_stats.checksum_errs: 9902
dev.ix.1.mac_stats.management_pkts_drpd: 0
dev.ix.1.mac_stats.management_pkts_rcvd: 0
dev.ix.1.mac_stats.recv_jabberd: 0
dev.ix.1.mac_stats.recv_oversized: 0
dev.ix.1.mac_stats.recv_fragmented: 0
dev.ix.1.mac_stats.recv_undersized: 0
dev.ix.1.mac_stats.rx_frames_1024_1522: 1656044
dev.ix.1.mac_stats.rx_frames_512_1023: 797853
dev.ix.1.mac_stats.rx_frames_256_511: 1043221
dev.ix.1.mac_stats.rx_frames_128_255: 8148642
dev.ix.1.mac_stats.rx_frames_65_127: 3508997
dev.ix.1.mac_stats.rx_frames_64: 1502805
dev.ix.1.mac_stats.bcast_pkts_rcvd: 779246
dev.ix.1.mac_stats.mcast_pkts_rcvd: 47489
dev.ix.1.mac_stats.good_pkts_rcvd: 16657562
dev.ix.1.mac_stats.total_pkts_rcvd: 16657563
dev.ix.1.mac_stats.good_octets_rcvd: 5610262402
dev.ix.1.mac_stats.total_octets_rcvd: 5610262402
dev.ix.1.mac_stats.xoff_recvd: 0
dev.ix.1.mac_stats.xoff_txd: 0
dev.ix.1.mac_stats.xon_recvd: 0
dev.ix.1.mac_stats.xon_txd: 0
dev.ix.1.mac_stats.rx_missed_packets: 0
dev.ix.1.mac_stats.rec_len_errs: 0
dev.ix.1.mac_stats.remote_faults: 0
dev.ix.1.mac_stats.local_faults: 10
dev.ix.1.mac_stats.short_discards: 0
dev.ix.1.mac_stats.byte_errs: 0
dev.ix.1.mac_stats.ill_errs: 0
dev.ix.1.mac_stats.crc_errs: 0
dev.ix.1.mac_stats.rx_errs: 9902
[dd@fw2 dd]$ sysctl dev.ix.1.mac_stats
dev.ix.1.mac_stats.tx_frames_1024_1522: 11685
dev.ix.1.mac_stats.tx_frames_512_1023: 5965
dev.ix.1.mac_stats.tx_frames_256_511: 10170
dev.ix.1.mac_stats.tx_frames_128_255: 113592
dev.ix.1.mac_stats.tx_frames_65_127: 125954
dev.ix.1.mac_stats.tx_frames_64: 4238
dev.ix.1.mac_stats.management_pkts_txd: 0
dev.ix.1.mac_stats.mcast_pkts_txd: 263
dev.ix.1.mac_stats.bcast_pkts_txd: 118
dev.ix.1.mac_stats.good_pkts_txd: 271604
dev.ix.1.mac_stats.total_pkts_txd: 271604
dev.ix.1.mac_stats.good_octets_txd: 56640166
dev.ix.1.mac_stats.checksum_errs: 1216
dev.ix.1.mac_stats.management_pkts_drpd: 0
dev.ix.1.mac_stats.management_pkts_rcvd: 0
dev.ix.1.mac_stats.recv_jabberd: 0
dev.ix.1.mac_stats.recv_oversized: 0
dev.ix.1.mac_stats.recv_fragmented: 0
dev.ix.1.mac_stats.recv_undersized: 0
dev.ix.1.mac_stats.rx_frames_1024_1522: 4146
dev.ix.1.mac_stats.rx_frames_512_1023: 14575
dev.ix.1.mac_stats.rx_frames_256_511: 10745
dev.ix.1.mac_stats.rx_frames_128_255: 36205
dev.ix.1.mac_stats.rx_frames_65_127: 334569
dev.ix.1.mac_stats.rx_frames_64: 823070
dev.ix.1.mac_stats.bcast_pkts_rcvd: 783535
dev.ix.1.mac_stats.mcast_pkts_rcvd: 138032
dev.ix.1.mac_stats.good_pkts_rcvd: 1223310
dev.ix.1.mac_stats.total_pkts_rcvd: 1223310
dev.ix.1.mac_stats.good_octets_rcvd: 110608648
dev.ix.1.mac_stats.total_octets_rcvd: 110608648
dev.ix.1.mac_stats.xoff_recvd: 0
dev.ix.1.mac_stats.xoff_txd: 0
dev.ix.1.mac_stats.xon_recvd: 0
dev.ix.1.mac_stats.xon_txd: 0
dev.ix.1.mac_stats.rx_missed_packets: 0
dev.ix.1.mac_stats.rec_len_errs: 0
dev.ix.1.mac_stats.remote_faults: 0
dev.ix.1.mac_stats.local_faults: 10
dev.ix.1.mac_stats.short_discards: 0
dev.ix.1.mac_stats.byte_errs: 0
dev.ix.1.mac_stats.ill_errs: 0
dev.ix.1.mac_stats.crc_errs: 0
dev.ix.1.mac_stats.rx_errs: 1216
The upgrade from 12.2-STABLE to 12.4-RELEASE was the only change made when the
errors began to appear. Everything else remained the same -- the physical layer
(network cables, switch, etc.), the monitoring check plugin, the switch config,
the rc.conf config, etc.
--
You are receiving this mail because:
You are the assignee for the bug.