Too much interrupts on ixgbe
Andrew Boyer
aboyer at averesystems.com
Mon Oct 24 20:10:10 UTC 2011
You could try this patch. It disables the interrupt while it's being handled. (The driver already re-enables it at the end of the handler and the task.)
-Andrew
Index: sys/dev/ixgbe/ixgbe.c
===================================================================
--- sys/dev/ixgbe/ixgbe.c (revision 226698)
+++ sys/dev/ixgbe/ixgbe.c (working copy)
@@ -1362,6 +1362,7 @@
bool more_tx, more_rx;
u32 newitr = 0;
+ ixgbe_disable_queue(adapter, que->msix);
++que->irqs;
more_rx = ixgbe_rxeof(que, adapter->rx_process_limit);
On Oct 24, 2011, at 5:41 AM, Sergey Saley wrote:
> There is my FreeBSD box:
>
> kernel
> ---------------
> #
> # GENERIC -- Generic kernel configuration file for FreeBSD/i386
> #
> # For more information on this file, please read the config(5) manual page,
> # and/or the handbook section on Kernel Configuration Files:
> #
> #
> http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
> #
> # The handbook is also available locally in /usr/share/doc/handbook
> # if you've installed the doc distribution, otherwise always see the
> # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
> # latest information.
> #
> # An exhaustive list of options and more detailed explanations of the
> # device lines is also present in the ../../conf/NOTES and NOTES files.
> # If you are in doubt as to the purpose or necessity of a line, check first
> # in NOTES.
> #
> # $FreeBSD: head/sys/i386/conf/GENERIC 221743 2011-05-10 16:44:16Z jkim $
>
> cpu I686_CPU
> ident POINT07
>
>
> options SCHED_ULE # ULE scheduler
> options PREEMPTION # Enable kernel thread preemption
> options INET # InterNETworking
> options FFS # Berkeley Fast Filesystem
> options SOFTUPDATES # Enable FFS soft updates support
> options UFS_DIRHASH # Improve performance on big directories
> options MD_ROOT # MD is a potential root device
> options MSDOSFS # MSDOS Filesystem
> options CD9660 # ISO 9660 Filesystem
> options PROCFS # Process filesystem (requires PSEUDOFS)
> options PSEUDOFS # Pseudo-filesystem framework
> options GEOM_PART_GPT # GUID Partition Tables.
> options GEOM_LABEL # Provides labelization
> options KTRACE # ktrace(1) support
> options STACK # stack(9) support
> options SYSVSHM # SYSV-style shared memory
> options SYSVMSG # SYSV-style message queues
> options SYSVSEM # SYSV-style semaphores
> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
> options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed.
> options KBD_INSTALL_CDEV # install a CDEV entry in /dev
> options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4)
> #options KDTRACE_HOOKS # Kernel DTrace hooks
> options INCLUDE_CONFIG_FILE # Include this file in kernel
>
>
> # To make an SMP kernel, the next two lines are needed
> options SMP # Symmetric MultiProcessor Kernel
> device apic # I/O APIC
>
> # CPU frequency control
> device cpufreq
>
> # Bus support.
> device acpi
> device pci
>
> # Floppy drives
> device fdc
>
> # ATA controllers
> device ahci # AHCI-compatible SATA controllers
> device ata # Legacy ATA/SATA controllers
> options ATA_CAM # Handle legacy controllers with CAM
> options ATA_STATIC_ID # Static device numbering
> device mvs # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA
> device siis # SiliconImage SiI3124/SiI3132/SiI3531 SATA
>
> # ATA/SCSI peripherals
> device scbus # SCSI bus (required for ATA/SCSI)
> device ch # SCSI media changers
> device da # Direct Access (disks)
> device sa # Sequential Access (tape etc)
> device cd # CD
> device pass # Passthrough device (direct ATA/SCSI access)
> device ses # SCSI Environmental Services (and SAF-TE)
>
>
> # atkbdc0 controls both the keyboard and the PS/2 mouse
> device atkbdc # AT keyboard controller
> device atkbd # AT keyboard
> device psm # PS/2 mouse
>
> device kbdmux # keyboard multiplexer
>
> device vga # VGA video card driver
>
>
> # syscons is the default console driver, resembling an SCO console
> device sc
> options SC_PIXEL_MODE # add support for the raster text mode
>
> device agp # support several AGP chipsets
>
> # Power management support (see NOTES for more options)
> #device apm
> # Add suspend/resume support for the i8254.
> device pmtimer
>
> # Serial (COM) ports
> device uart # Generic UART driver
>
> # Parallel port
> device ppc
> device ppbus # Parallel port bus (required)
> device lpt # Printer
> #device plip # TCP/IP over parallel
> device ppi # Parallel port interface device
> #device vpo # Requires scbus and da
>
> # If you've got a "dumb" serial or parallel PCI card that is
> # supported by the puc(4) glue driver, uncomment the following
> # line to enable it (connects to sio, uart and/or ppc drivers):
> #device puc
>
> # PCI Ethernet NICs.
> device em # Intel PRO/1000 Gigabit Ethernet Family
> device ixgbe # Intel PRO/10GbE Ethernet Card
>
> # Pseudo devices.
> device loop # Network loopback
> device random # Entropy device
> device ether # Ethernet support
> device vlan # 802.1Q VLAN support
> #device tun # Packet tunnel.
> device pty # BSD-style compatibility pseudo ttys
> device md # Memory "disks"
> device gif # IPv6 and IPv4 tunneling
> device faith # IPv6-to-IPv4 relaying (translation)
> device firmware # firmware assist module
>
> # The `bpf' device enables the Berkeley Packet Filter.
> # Be aware of the administrative consequences of enabling this!
> # Note that 'bpf' is required for DHCP.
> device bpf # Berkeley packet filter
>
>
> options IPFIREWALL
> options IPFIREWALL_DEFAULT_TO_ACCEPT
> options IPFIREWALL_FORWARD
> options DUMMYNET
> options IPI_PREEMPTION
>
> # netgraph(4). Enable the base netgraph code with the NETGRAPH option.
> # Individual node types can be enabled with the corresponding option
> # listed below; however, this is not strictly necessary as netgraph
> # will automatically load the corresponding KLD module if the node type
> # is not already compiled into the kernel. Each type below has a
> # corresponding man page, e.g., ng_async(8).
>
> options NETGRAPH # netgraph(4) system
>
> options NETGRAPH_CAR
> options NETGRAPH_IFACE
> options NETGRAPH_MPPC_ENCRYPTION
> options NETGRAPH_PPP
> options NETGRAPH_PPPOE
> options NETGRAPH_PPTPGRE
> options NETGRAPH_SOCKET
> options NETGRAPH_TCPMSS
> options NETGRAPH_ETHER
> options NETGRAPH_TEE
> options NETGRAPH_VJC
> options NETGRAPH_BPF
> options NETGRAPH_KSOCKET
> options NETGRAPH_IPFW
>
> dmesg.boot
> --------------------------
> Copyright (c) 1992-2011 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 9.0-BETA3 #1: Fri Oct 21 10:37:43 FET 2011
> root at point07.uch.net:/usr/obj/usr/src/sys/POINT07 amd64
> CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2394.05-MHz K8-class
> CPU)
> Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11
>
> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
> AMD Features=0x20100800<SYSCALL,NX,LM>
> AMD Features2=0x1<LAHF>
> TSC: P-state invariant, performance statistics
> real memory = 2147483648 (2048 MB)
> avail memory = 2056187904 (1960 MB)
> Event timer "LAPIC" quality 400
> ACPI APIC Table: <PTLTD APIC >
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> FreeBSD/SMP: 1 package(s) x 4 core(s)
> cpu0 (BSP): APIC ID: 0
> cpu1 (AP): APIC ID: 1
> cpu2 (AP): APIC ID: 2
> cpu3 (AP): APIC ID: 3
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> kbd1 at kbdmux0
> acpi0: <PTLTD XSDT> on motherboard
> acpi0: Power Button (fixed)
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
> cpu0: <ACPI CPU> on acpi0
> cpu1: <ACPI CPU> on acpi0
> cpu2: <ACPI CPU> on acpi0
> cpu3: <ACPI CPU> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11> port
> 0x2000-0x201f mem 0xdc680000-0xdc6fffff,0xdc600000-0xdc603fff irq 16 at
> device 0.0 on pci1
> ix0: Using MSIX interrupts with 5 vectors
> ix0: Ethernet address: 00:25:90:3c:42:6a
> ix0: PCI Express Bus: Speed 2.5Gb/s Width x8
> ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11> port
> 0x2020-0x203f mem 0xdc700000-0xdc77ffff,0xdc604000-0xdc607fff irq 17 at
> device 0.1 on pci1
> ix1: Using MSIX interrupts with 5 vectors
> ix1: RX Descriptors exceed system mbuf max, using default instead!
> ix1: Ethernet address: 00:25:90:3c:42:6b
> ix1: PCI Express Bus: Speed 2.5Gb/s Width x8
> pci0: <serial bus, USB> at device 26.0 (no driver attached)
> pci0: <serial bus, USB> at device 26.1 (no driver attached)
> pci0: <serial bus, USB> at device 26.2 (no driver attached)
> pci0: <serial bus, USB> at device 26.7 (no driver attached)
> pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
> pci5: <ACPI PCI bus> on pcib2
> pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
> pci13: <ACPI PCI bus> on pcib3
> em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x3000-0x301f mem
> 0xdc100000-0xdc11ffff irq 16 at device 0.0 on pci13
> em0: Using an MSI interrupt
> em0: Ethernet address: 00:30:48:d2:12:ac
> pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
> pci15: <ACPI PCI bus> on pcib4
> em1: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x4000-0x401f mem
> 0xdc200000-0xdc21ffff irq 17 at device 0.0 on pci15
> em1: Using an MSI interrupt
> em1: Ethernet address: 00:30:48:d2:12:ad
> pci0: <serial bus, USB> at device 29.0 (no driver attached)
> pci0: <serial bus, USB> at device 29.1 (no driver attached)
> pci0: <serial bus, USB> at device 29.2 (no driver attached)
> pci0: <serial bus, USB> at device 29.7 (no driver attached)
> pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci17: <ACPI PCI bus> on pcib5
> vgapci0: <VGA-compatible display> port 0x5000-0x507f mem
> 0xde000000-0xdfffffff,0xdc300000-0xdc33ffff at device 4.0 on pci17
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel ICH9 SATA300 controller> port
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1c10-0x1c1f,0x1c00-0x1c0f at device
> 31.2 on pci0
> ata0: <ATA channel 0> on atapci0
> ata1: <ATA channel 1> on atapci0
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> atapci1: <Intel ICH9 SATA300 controller> port
> 0x1c68-0x1c6f,0x1c5c-0x1c5f,0x1c60-0x1c67,0x1c58-0x1c5b,0x1c30-0x1c3f,0x1c20-0x1c2f
> irq 18 at device 31.5 on pci0
> ata2: <ATA channel 0> on atapci1
> ata3: <ATA channel 1> on atapci1
> pci0: <dasp> at device 31.6 (no driver attached)
> acpi_button0: <Power Button> on acpi0
> atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
> Event timer "RTC" frequency 32768 Hz quality 0
> attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
> ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> ppc0: FIFO with 16/16/9 bytes threshold
> ppbus0: <Parallel port bus> on ppc0
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> est2: <Enhanced SpeedStep Frequency Control> on cpu2
> p4tcc2: <CPU Frequency Thermal Control> on cpu2
> est3: <Enhanced SpeedStep Frequency Control> on cpu3
> p4tcc3: <CPU Frequency Thermal Control> on cpu3
> Timecounters tick every 1.000 msec
> ipfw2 initialized, divert loadable, nat loadable, rule-based forwarding
> enabled, default to accept, logging disabled
> DUMMYNET 0 with IPv6 initialized (100409)
> load_dn_sched dn_sched QFQ loaded
> load_dn_sched dn_sched RR loaded
> load_dn_sched dn_sched WF2Q+ loaded
> load_dn_sched dn_sched FIFO loaded
> load_dn_sched dn_sched PRIO loaded
> ada0 at ata0 bus 0 scbus0 target 0 lun 0
> ada0: <SAMSUNG HD161HJ GF100-07> ATA-8 SATA 2.x device
> ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
> ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C)
> ada0: Previously was known as ad0
> SMP: AP CPU #3 Launched!
> SMP: AP CPU #1 Launched!
> SMP: AP CPU #2 Launched!
> Timecounter "TSC-low" frequency 9351750 Hz quality 1000
> Trying to mount root from ufs:/dev/ada0p2 [rw]...
>
> sysctl.conf
> --------------
> net.inet.ip.fastforwarding=1
> net.isr.direct=0
> net.isr.direct_force=0
> net.inet.ip.fw.one_pass=1
> hw.intr_storm_threshold=9000
> kern.ipc.nmbclusters=262144
> kern.ipc.nmbjumbop=262144
> dev.ix.0.rx_processing_limit=4096
> dev.ix.1.rx_processing_limit=4096
> net.inet.ip.intr_queue_maxlen=256
> dev.ix.0.flow_control=0
> dev.ix.1.flow_control=0
>
> vmstat -i
> -------------
> # vmstat -i
> interrupt total rate
> irq1: atkbd0 1 0
> irq6: fdc0 1 0
> irq14: ata0 80855 9
> irq15: ata1 1 0
> cpu0:timer 9694291 1122
> irq256: ix0:que 0 54169187 6273
> irq257: ix0:que 1 19307922 2236
> irq258: ix0:que 2 24265573 2810
> irq259: ix0:que 3 25845865 2993
> irq260: ix0:link 2 0
> irq261: ix1:que 0 63210220 7321
> irq262: ix1:que 1 265895 30
> irq263: ix1:que 2 42114 4
> irq264: ix1:que 3 98953 11
> irq265: ix1:link 5 0
> cpu3:timer 7165968 829
> cpu1:timer 7386880 855
> cpu2:timer 7307920 846
> Total 218841653 25346
>
> netstat
> ------------
> # netstat -I ix0 -w1
> input (ix0) output
> packets errs idrops bytes packets errs bytes colls
> 26592 0 0 30220498 20757 0 11490639 0
> 31232 0 0 34001150 24699 0 14554631 0
> 27785 0 0 29672129 23699 0 16154542 0
> 25525 0 0 27450183 22868 0 16180216 0
> ^C
> # netstat -I ix1 -w1
> input (ix1) output
> packets errs idrops bytes packets errs bytes colls
> 22398 0 0 17232635 23781 0 24718205 0
> 23671 0 0 17608589 25859 0 28970777 0
> 24096 0 0 17278557 27188 0 30804019 0
> 22043 0 0 16271250 24116 0 26238423 0
> ^C
>
>
> top -SCHP
> -----------------
> last pid: 74628; load averages: 1.26, 1.01, 0.89
> up 0+02:25:14 13:22:11
> 110 processes: 9 running, 79 sleeping, 22 waiting
> CPU 0: 0.4% user, 0.0% nice, 3.1% system, 44.1% interrupt, 52.4% idle
> CPU 1: 0.0% user, 0.0% nice, 3.1% system, 13.4% interrupt, 83.5% idle
> CPU 2: 0.8% user, 0.0% nice, 2.4% system, 18.5% interrupt, 78.3% idle
> CPU 3: 0.4% user, 0.0% nice, 4.7% system, 13.0% interrupt, 81.9% idle
> Mem: 44M Active, 118M Inact, 326M Wired, 792K Cache, 213M Buf, 1487M Free
> Swap: 4096M Total, 4096M Free
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND
> 10 root 155 ki31 0K 64K RUN 2 130:32 94.68% idle{idle:
> cpu2}
> 10 root 155 ki31 0K 64K RUN 1 130:49 93.07% idle{idle:
> cpu1}
> 10 root 155 ki31 0K 64K CPU3 3 132:48 89.26% idle{idle:
> cpu3}
> 10 root 155 ki31 0K 64K RUN 0 101:00 58.89% idle{idle:
> cpu0}
> 11 root -92 - 0K 416K CPU0 0 30:52 36.77% intr{irq261:
> ix1:que }
> 11 root -92 - 0K 416K CPU3 3 8:03 11.77% intr{irq259:
> ix0:que }
> 11 root -92 - 0K 416K CPU1 1 5:37 9.86% intr{irq257:
> ix0:que }
> 11 root -92 - 0K 416K WAIT 2 8:14 9.28% intr{irq258:
> ix0:que }
> 11 root -92 - 0K 416K RUN 0 7:07 5.18% intr{irq256:
> ix0:que }
> 0 root -92 0 0K 352K - 2 1:12 1.17% kernel{ix1
> que}
> 12 root -16 - 0K 64K sleep 0 1:02 1.07%
> ng_queue{ng_queue1}
> 12 root -16 - 0K 64K sleep 0 1:02 0.88%
> ng_queue{ng_queue0}
> 12 root -16 - 0K 64K sleep 2 1:02 0.78%
> ng_queue{ng_queue3}
> 12 root -16 - 0K 64K sleep 0 1:02 0.78%
> ng_queue{ng_queue2}
> 0 root -92 0 0K 352K - 3 0:58 0.39% kernel{ix0
> que}
> 11 root -60 - 0K 416K WAIT 3 5:21 0.00% intr{swi4:
> clock}
> 0 root -16 0 0K 352K sched 1 0:54 0.00%
> kernel{swapper}
> 14320 root 20 0 78528K 32900K select 1 0:52 0.00% mpd5{mpd5}
> 0 root -92 0 0K 352K - 0 0:28 0.00% kernel{ix0
> que}
> 0 root -92 0 0K 352K - 2 0:25 0.00%
> kernel{dummynet}
> 2382 root 20 0 18832K 4060K select 3 0:21 0.00% zebra
> 0 root -92 0 0K 352K - 2 0:17 0.00% kernel{ix0
> que}
> 14 root -16 - 0K 16K - 2 0:12 0.00% yarrow
> 0 root -92 0 0K 352K - 3 0:07 0.00% kernel{ix0
> que}
> 2388 root 20 0 25632K 6900K select 3 0:06 0.00% ospfd
> 19419 www 20 0 16332K 5440K kqread 1 0:05 0.00% thttpd
> 3 root -16 - 0K 16K ccb_sc 0 0:04 0.00% xpt_thrd
> 9 root 16 - 0K 16K syncer 2 0:02 0.00% syncer
> 0 root -92 0 0K 352K - 1 0:02 0.00% kernel{ix1
> linkq}
> 3846 root 20 0 22340K 3452K select 1 0:02 0.00% ntpd
> 4005 root 20 0 68024K 5976K select 1 0:01 0.00% sshd
> 11 root -92 - 0K 416K WAIT 1 0:01 0.00% intr{irq262:
> ix1:que }
> 11 root -68 - 0K 416K WAIT 2 0:01 0.00% intr{swi2:
> cambio}
> 13 root -8 - 0K 48K - 3 0:01 0.00% geom{g_up}
> 57732 root 20 0 32136K 2920K uwait 1 0:01 0.00%
> collectd{collectd}
> 11 root -88 - 0K 416K WAIT 3 0:01 0.00% intr{irq14:
> ata0}
> 3576 root 20 0 12192K 1716K select 2 0:01 0.00% syslogd
> 81248 root 20 0 68024K 5708K select 1 0:01 0.00% sshd
> 13 root -8 - 0K 48K - 0 0:01 0.00% geom{g_down}
> 11 root -92 - 0K 416K WAIT 3 0:01 0.00% intr{irq264:
> ix1:que }
> 15 root -16 - 0K 16K sdflus 3 0:00 0.00% softdepflush
> 11 root -60 - 0K 416K WAIT 2 0:00 0.00% intr{swi4:
> clock}
>
> pciconf -lvc
> ---------------
> ix0 at pci0:1:0:0: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01
> hdr=0x00
> vendor = 'Intel Corporation'
> device = '82599EB 10-Gigabit SFI/SFP+ Network Connection'
> class = network
> subclass = ethernet
> cap 01[40] = powerspec 3 supports D0 D3 current D0
> cap 05[50] = MSI supports 1 message, 64 bit, vector masks
> cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled
> cap 10[a0] = PCI-Express 2 endpoint max data 128(512) link x8(x8)
> cap 03[e0] = VPD
> ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
> ecap 0003[140] = Serial 1 002590ffff3c426a
> ecap 000e[150] = unknown 1
> ecap 0010[160] = unknown 1
> ix1 at pci0:1:0:1: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01
> hdr=0x00
> vendor = 'Intel Corporation'
> device = '82599EB 10-Gigabit SFI/SFP+ Network Connection'
> class = network
> subclass = ethernet
> cap 01[40] = powerspec 3 supports D0 D3 current D0
> cap 05[50] = MSI supports 1 message, 64 bit, vector masks
> cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled
> cap 10[a0] = PCI-Express 2 endpoint max data 128(512) link x8(x8)
> cap 03[e0] = VPD
> ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
> ecap 0003[140] = Serial 1 002590ffff3c426a
> ecap 000e[150] = unknown 1
> ecap 0010[160] = unknown 1
>
> The question is:
> -----------------
> Why there are so much interrupts?
> Why the card is using only one intr on second port (ix1)?
>
> I've got another server with 82598EB 10 Gigabit AT CX4 Network Connection
> and the situation on it is dramatically different:
>
> last pid: 42234; load averages: 2.14, 1.65, 1.55
> up 119+05:58:58 13:26:01
> 98 processes: 8 running, 62 sleeping, 28 waiting
> CPU 0: 0.0% user, 0.0% nice, 0.0% system, 34.0% interrupt, 66.0% idle
> CPU 1: 0.0% user, 0.0% nice, 0.0% system, 43.4% interrupt, 56.6% idle
> CPU 2: 0.0% user, 0.0% nice, 0.0% system, 39.6% interrupt, 60.4% idle
> CPU 3: 0.0% user, 0.0% nice, 0.0% system, 35.8% interrupt, 64.2% idle
> Mem: 279M Active, 294M Inact, 300M Wired, 132K Cache, 112M Buf, 2128M Free
> Swap: 4096M Total, 4096M Free
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND
> 10 root 171 ki31 0K 32K RUN 0 2159.3 70.46% {idle: cpu0}
> 10 root 171 ki31 0K 32K RUN 2 2166.8 69.48% {idle: cpu2}
> 10 root 171 ki31 0K 32K RUN 3 2176.3 68.90% {idle: cpu3}
> 10 root 171 ki31 0K 32K RUN 1 2163.1 68.55% {idle: cpu1}
> 11 root -68 - 0K 248K CPU3 3 335.0H 19.38% {irq259:
> ix0:que }
> 11 root -68 - 0K 248K WAIT 1 335.5H 17.97% {irq257:
> ix0:que }
> 11 root -68 - 0K 248K CPU2 2 335.3H 17.77% {irq258:
> ix0:que }
> 11 root -68 - 0K 248K CPU0 0 336.2H 17.58% {irq256:
> ix0:que }
> 11 root -68 - 0K 248K WAIT 3 323.4H 17.09% {irq264:
> ix1:que }
> 11 root -68 - 0K 248K WAIT 1 323.9H 16.55% {irq262:
> ix1:que }
> 11 root -68 - 0K 248K WAIT 2 325.6H 16.06% {irq263:
> ix1:que }
> 11 root -68 - 0K 248K WAIT 0 325.9H 15.77% {irq261:
> ix1:que }
>
> vmstat -i
> -------------
> r# vmstat -i
> interrupt total rate
> irq19: atapci0 1667110 0
> cpu0: timer 3386721810 328
> irq256: ix0:que 0 3395843376 329
> irq257: ix0:que 1 2642665824 256
> irq258: ix0:que 2 2838302235 275
> irq259: ix0:que 3 2176207954 211
> irq260: ix0:link 18 0
> irq261: ix1:que 0 282359321 27
> irq262: ix1:que 1 3989170496 387
> irq263: ix1:que 2 375956573 36
> irq264: ix1:que 3 3966352151 384
> irq265: ix1:link 1 0
> irq266: em0:rx 0 10283114 0
> irq269: em1:rx 0 10283114 0
> cpu3: timer 3386697130 328
> cpu1: timer 3386709028 328
> cpu2: timer 3386700908 328
> Total 33235920163 3225
>
> netstat
> ------------
> # netstat -I ix0 -w1
> input (ix0) output
> packets errs idrops bytes packets errs bytes colls
> 264572 0 0 252383894 253290 0 178845781 0
> 266323 0 0 254825280 251277 0 173769218 0
> 274462 0 0 265247306 260819 0 181113304 0
> 271142 0 0 263263325 253941 0 171694438 0
> ^C
> # netstat -I ix1 -w1
> input (ix1) output
> packets errs idrops bytes packets errs bytes colls
> 259427 0 0 183038665 275711 0 262429749 0
> 248177 0 0 171479966 264441 0 252993752 0
> 255271 0 0 185006000 266886 0 247494148 0
> 264543 0 0 190970875 275087 0 259555066 0
> ^C
>
> Tuning on both systems are almost the same
> As You can see, 82598EB card produce about 20 times less interrupts at about
> 10 times more pps.
>
> Please help me to fix the problem...
>
> --
> View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4931883.html
> Sent from the freebsd-net mailing list archive at Nabble.com.
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
--------------------------------------------------
Andrew Boyer aboyer at averesystems.com
More information about the freebsd-net
mailing list