[Bug 280386] if_bridge throws output errors under load
Date: Sun, 21 Jul 2024 04:53:33 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280386
Bug ID: 280386
Summary: if_bridge throws output errors under load
Product: Base System
Version: 14.1-RELEASE
Hardware: Any
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: bugs@FreeBSD.org
Reporter: pascal.guitierrez@gmail.com
I'm able to easily make if_bridge spew thousands of output errors with a really
simple config and using iperf3.
this results into random "no buffer space" errors on the host and disruptions
to services.
only reproduceable when using a bridge - if no bridge involved there are no
errors or disruptions.
OS is 14.1-RELEASE-p2 on amd64.
/etc/rc.conf:
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 192.168.0.150/24 up"
ifconfig_igb0="-txcsum -txcsum6 -tso up"
(the reason for turning off certain flags is to prevent link-flap when the VM
is disconnected from the bridge)
netstat -i:
(note the Oerrs field below)
Name Mtu Network Address Ipkts Ierrs Idrop Opkts
Oerrs Coll
igb0 1500 <Link#1> 5c:ed:8c:e9:c2:48 50259033 0 0 53166489
0 0
igb1* 1500 <Link#2> 5c:ed:8c:e9:c2:49 0 0 0 0
0 0
igb2* 1500 <Link#3> 5c:ed:8c:e9:c2:4a 0 0 0 0
0 0
igb3* 1500 <Link#4> 5c:ed:8c:e9:c2:4b 0 0 0 0
0 0
lo0 16384 <Link#5> lo0 24959 0 0 24959
0 0
lo0 - localhost localhost 0 - - 0
- -
lo0 - fe80::%lo0/64 fe80::1%lo0 0 - - 0
- -
lo0 - your-net localhost 24959 - - 24959
- -
bridge0 1500 <Link#6> 58:9c:fc:00:07:00 103262907 0 0 103141654
130543 0
bridge0 - 192.168.0.0/24 192.168.0.150 235137 - - 225757
- -
ue0 1500 <Link#7> 72:84:d1:bf:ad:2f 3587 0 0 3588
0 0
ue0 - 16.1.15.0/30 16.1.15.2 3308 - - 3308
- -
tap0 1500 <Link#8> 58:9c:fc:10:f3:0f 53003359 0 0 48249364
0 0
can reproduce across multiple systems of varying compute and network power
(have tested on ix* NICs with similar results)
iperf3 cmd line used:
iperf3 -P4 -c 192.168.0.5 -t 60
remote iperf3 system is a windows server, network is gigabit.
errors occur only when testing in one direction (FreeBSD bridge client ->
remote system server).
if i don't use if_bridge, ie:
ifconfig_igb0="-txcsum -txcsum6 -tso 192.168.0.150/24 up"
then no errors or dropped packets are seen at all.
is my config correct? are there known issues with if_bridge in freebsd?
dmesg below:
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git
llvmorg-18.1.5-0-g617a15a9eac9)
VT(efifb): resolution 1024x768
CPU: Intel(R) Xeon(R) E-2314 CPU @ 2.80GHz (2808.00-MHz K8-class CPU)
Origin="GenuineIntel" Id=0xa0671 Family=0x6 Model=0xa7 Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended
Features=0xf2bf67ef<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,PROCTRACE,AVX512CD,SHA,AVX512BW,AVX512VL>
Structured Extended
Features2=0x40405f5e<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,AVX512VPOPCNTDQ,RDPID,SGXLC>
Structured Extended
Features3=0xbc000410<FSRM,MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
IA32_ARCH_CAPS=0x2023c6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 17179869184 (16384 MB)
avail memory = 16503263232 (15738 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HPE Server2 >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 2 3 1
random: entropy device external interface
kbd1 at kbdmux0
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x72fc8000-0x72fc801e
smbios0: Version: 3.3, BCD Revision: 3.3
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <HPE Server2>
acpi0: Power Button (fixed)
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0
apei0: <ACPI Platform Error Interface> on acpi0
acpi_syscontainer0: <System Container> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
xhci0: <Intel Tiger Lake-H USB 3.2 controller> mem 0x4000000000-0x400000ffff at
device 20.0 numa-domain 0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 numa-domain 0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.4 (no driver attached)
ahci0: <AHCI SATA controller> port 0x3040-0x3047,0x3048-0x304b,0x3020-0x303f
mem 0x80580000-0x80581fff,0x80584000-0x805840ff,0x80583000-0x805837ff at device
23.0 numa-domain 0 on pci0
ahci0: AHCI v1.31 with 4 6Gbps ports, Port Multiplier not supported
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib1: <ACPI PCI-PCI bridge> at device 27.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
igb0: <Intel(R) I350 (Copper)> port 0x2060-0x207f mem
0x80300000-0x803fffff,0x8040c000-0x8040ffff at device 0.0 numa-domain 0 on pci1
igb0: EEPROM V1.63-0 Option ROM V1-b3310-p0 eTrack 0x8000119d
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
igb0: Ethernet address: 5c:ed:8c:e9:c2:48
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: <Intel(R) I350 (Copper)> port 0x2040-0x205f mem
0x80200000-0x802fffff,0x80408000-0x8040bfff at device 0.1 numa-domain 0 on pci1
igb1: EEPROM V1.63-0 Option ROM V1-b3310-p0 eTrack 0x8000119d
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
igb1: Ethernet address: 5c:ed:8c:e9:c2:49
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
igb2: <Intel(R) I350 (Copper)> port 0x2020-0x203f mem
0x80100000-0x801fffff,0x80404000-0x80407fff at device 0.2 numa-domain 0 on pci1
igb2: EEPROM V1.63-0 Option ROM V1-b3310-p0 eTrack 0x8000119d
igb2: Using 1024 TX descriptors and 1024 RX descriptors
igb2: Using 4 RX queues 4 TX queues
igb2: Using MSI-X interrupts with 5 vectors
igb2: Ethernet address: 5c:ed:8c:e9:c2:4a
igb2: netmap queues/slots: TX 4/1024, RX 4/1024
igb3: <Intel(R) I350 (Copper)> port 0x2000-0x201f mem
0x80000000-0x800fffff,0x80400000-0x80403fff at device 0.3 numa-domain 0 on pci1
igb3: EEPROM V1.63-0 Option ROM V1-b3310-p0 eTrack 0x8000119d
igb3: Using 1024 TX descriptors and 1024 RX descriptors
igb3: Using 4 RX queues 4 TX queues
igb3: Using MSI-X interrupts with 5 vectors
igb3: Ethernet address: 5c:ed:8c:e9:c2:4b
igb3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib2: <ACPI PCI-PCI bridge> at device 27.6 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 29.0 numa-domain 0 on pci0
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
vgapci0: <VGA-compatible display> mem
0xa4000000-0xa4ffffff,0xa5b98000-0xa5b9bfff,0xa5000000-0xa57fffff at device 0.1
numa-domain 0 on pci3
vgapci0: Boot video device
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xa5b9c000-0xa5b9c0ff at device
0.4 numa-domain 0 on pci3
usbus1: EHCI version 1.0
usbus1 numa-domain 0 on ehci0
usbus1: 480Mbps High Speed USB v2.0
pcib4: <ACPI PCI-PCI bridge> at device 29.1 numa-domain 0 on pci0
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 29.2 numa-domain 0 on pci0
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 29.3 numa-domain 0 on pci0
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 29.4 numa-domain 0 on pci0
pci7: <ACPI PCI bus> numa-domain 0 on pcib7
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
pci0: <serial bus> at device 31.5 (no driver attached)
cpu0: <ACPI CPU> numa-domain 0 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags
0x10 on acpi0
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff pnpid PNP0900
on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 15.
hwpstate_intel0: <Intel Speed Shift> numa-domain 0 on cpu0
hwpstate_intel1: <Intel Speed Shift> numa-domain 0 on cpu1
hwpstate_intel2: <Intel Speed Shift> numa-domain 0 on cpu2
hwpstate_intel3: <Intel Speed Shift> numa-domain 0 on cpu3
Timecounter "TSC-low" frequency 1404004245 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ugen1.1: <(0x103c) EHCI root HUB> at usbus1
Trying to mount root from zfs:zroot/ROOT/default []...
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 numa-domain 0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub1 numa-domain 0 on usbus1
uhub1: <(0x103c) EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ada0 at ahcich2 bus 0 scbus0 target 0 lun 0
ada0: <ST4000NM002A-2HZ101 SN03> ACS-4 ATA SATA 3.x device
ada0: Serial Number WS255C4V
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 3815447MB (7814037168 512 byte sectors)
ada1 at ahcich3 bus 0 scbus1 target 0 lun 0
ada1: <ST4000NM002A-2HZ101 SN03> ACS-4 ATA SATA 3.x device
ada1: Serial Number WS255BGF
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)
ada2 at ahcich4 bus 0 scbus2 target 0 lun 0
ada2: <INTEL SSDSC2BB240G6 G2010130> ACS-2 ATA SATA 3.x device
ada2: Serial Number BTWA518300MH240AGN
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada2: Command Queueing enabled
ada2: 228936MB (468862128 512 byte sectors)
ada3 at ahcich5 bus 0 scbus3 target 0 lun 0
ada3: <INTEL SSDSC2BB240G6 G2010130> ACS-2 ATA SATA 3.x device
ada3: Serial Number BTWA51740BM1240AGN
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada3: Command Queueing enabled
ada3: 228936MB (468862128 512 byte sectors)
ses0 at ahciem0 bus 0 scbus4 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses0: ada0,pass0 in 'Slot 02', SATA Slot: scbus0 target 0
ses0: ada1,pass1 in 'Slot 03', SATA Slot: scbus1 target 0
ses0: ada2,pass2 in 'Slot 04', SATA Slot: scbus2 target 0
ses0: ada3,pass3 in 'Slot 05', SATA Slot: scbus3 target 0
uhub0: 22 ports with 22 removable, self powered
Root mount waiting for: usbus0 usbus1
Root mount waiting for: usbus0 usbus1
usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device Generic
Mass Storage Device (0x14cd:0x1212)
usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device
Generic Mass Storage Device (0x14cd:0x1212)
usb_msc_auto_quirk: UQ_MSC_NO_SYNC_CACHE set for USB mass storage device
Generic Mass Storage Device (0x14cd:0x1212)
ugen0.2: <Generic Mass Storage Device> at usbus0
umass0 numa-domain 0 on uhub0
umass0: <Generic Mass Storage Device, class 0/0, rev 2.00/1.00, addr 1> on
usbus0
umass0: SCSI over Bulk-Only; quirks = 0xc100
umass0:5:0: Attached to scbus5
da0 at umass-sim0 bus 0 scbus5 target 0 lun 0
da0: <Mass Storage Device 1.00> Removable Direct Access SCSI device
da0: Serial Number 121220160204
da0: 40.000MB/s transfers
da0: 3968MB (8126464 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
uhub1: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0 usbus1
ugen1.2: <HPE Virtual NIC> at usbus1
uhub_explore: illegal enable change, port 4
usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device Generic
Mass Storage Device (0x14cd:0x1212)
usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device
Generic Mass Storage Device (0x14cd:0x1212)
usb_msc_auto_quirk: UQ_MSC_NO_SYNC_CACHE set for USB mass storage device
Generic Mass Storage Device (0x14cd:0x1212)
ugen0.3: <Generic Mass Storage Device> at usbus0
umass1 numa-domain 0 on uhub0
umass1: <Generic Mass Storage Device, class 0/0, rev 2.00/1.00, addr 2> on
usbus0
umass1: SCSI over Bulk-Only; quirks = 0xc100
umass1:6:1: Attached to scbus6
da1 at umass-sim1 bus 1 scbus6 target 0 lun 0
da1: <Mass Storage Device 1.00> Removable Direct Access SCSI device
da1: Serial Number 121220160204
da1: 40.000MB/s transfers
da1: 3968MB (8126464 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>
ugen0.4: <vendor 0x0424 product 0x2660> at usbus0
uhub2 numa-domain 0 on uhub0
uhub2: <vendor 0x0424 product 0x2660, class 9/0, rev 2.00/8.01, addr 3> on
usbus0
uhub2: 2 ports with 1 removable, self powered
ichsmb0: <Intel Tiger Lake SMBus controller> port 0xefa0-0xefbf mem
0x4000014000-0x40000140ff at device 31.4 numa-domain 0 on pci0
smbus0: <System Management Bus> numa-domain 0 on ichsmb0
driver bug: Unable to set devclass (class: atrtc devname: (unknown))
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: cannot find EC device
bridge0: Ethernet address: 58:9c:fc:00:07:00
igb0: link state changed to UP
lo0: link state changed to UP
igb0: link state changed to DOWN
bridge0: link state changed to UP
igb0: promiscuous mode enabled
cdceem0 numa-domain 0 on uhub1
cdceem0: <HPE Virtual NIC, class 2/12, rev 2.00/0.01, addr 2> on usbus1
ue0: <USB Ethernet> on cdceem0
ue0: Ethernet address: 72:84:d1:bf:ad:2f
igb0: link state changed to UP
driver bug: Unable to set devclass (class: atrtc devname: (unknown))
tap0: Ethernet address: 58:9c:fc:10:f3:0f
tap0: promiscuous mode enabled
tap0: link state changed to UP
driver bug: Unable to set devclass (class: atrtc devname: (unknown))
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
ipmi0: IPMI device rev. 3, firmware rev. 3.01, version 2.0, device support mask
0x9f
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
--
You are receiving this mail because:
You are the assignee for the bug.