ixgbe interface micro stalls / slow responses
Коньков Евгений
kes-kes at yandex.ru
Mon Mar 12 16:49:54 UTC 2012
Здравствуйте, Steven.
Вы писали 12 марта 2012 г., 17:05:12:
SH> We've got a machine where with an ix interface on 8.2-RELEASE
SH> which is seeing intermittent slow responses. It shows as stalls
SH> on the console and is visible as high pings on an mtr
SH> from a local machine e.g.
SH> Packets Pings
SH> Host Loss% Snt Last Avg Best Wrst StDev
SH> 1. X.X.X.X 0.0% 181 0.1 117.7 0.1 2665. 314.8
SH> We've tried updating from 2.3.10 release driver + alias fix
SH> to 2.4.5 (the latest from 8.3) but still the behavour is the
SH> same.
SH> If we do a trace to an igb on the same machine everything is
SH> clean.
SH> Packets Pings
SH> Host Loss% Snt Last Avg Best Wrst StDev
SH> 1. 10.10.10.64 0.0% 136 0.1 0.2 0.1 12.5 1.1
SH> We are seeing "RX Descriptors exceed system mbuf max, using
SH> default instead!" on boot with the latest driver but the fix
SH> listed in the readme has no effect, as in sysctl.conf we have
SH> kern.ipc.nmbclusters=524288
SH> kern.ipc.nmbjumbop=262144
SH> Nothing looks out of the ordinary by there's definitely a
SH> problem there somewhere, any ideas?
SH> Detailed info which may be use below.
>>From dmeg:-
SH> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5> port 0x2000-0x201f mem
SH> 0xd8400000-0xd847ffff,0xd8480000-0xd8483fff irq 52 at device 0.0 on pci5
SH> ix0: Using MSIX interrupts with 9 vectors
SH> ix0: RX Descriptors exceed system mbuf max, using default instead!
SH> ix0: [ITHREAD]
SH> ix0: [ITHREAD]
SH> ix0: [ITHREAD]
SH> ix0: [ITHREAD]
SH> ix0: [ITHREAD]
SH> ix0: [ITHREAD]
SH> ix0: [ITHREAD]
SH> ix0: [ITHREAD]
SH> ix0: [ITHREAD]
SH> ix0: Ethernet address: 00:1b:21:7e:2e:8c
SH> ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
SH> pciconf -v -l
SH> ix0 at pci0:5:0:0: class=0x020000 card=0x00068086 chip=0x10fb8086 rev=0x01 hdr=0x00
SH> vendor = 'Intel Corporation'
SH> class = network
SH> subclass = ethernet
SH> sysctl dev.ix
SH> dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5
SH> dev.ix.0.%driver: ix
SH> dev.ix.0.%location: slot=0 function=0
SH> dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10fb subvendor=0x8086 subdevice=0x0006 class=0x020000
SH> dev.ix.0.%parent: pci5
SH> dev.ix.0.fc: 3
SH> dev.ix.0.advertise_gig: 0
SH> dev.ix.0.enable_aim: 1
SH> dev.ix.0.advertise_speed: 0
SH> dev.ix.0.rx_processing_limit: 128
SH> dev.ix.0.dropped: 0
SH> dev.ix.0.mbuf_defrag_failed: 0
SH> dev.ix.0.no_tx_dma_setup: 0
SH> dev.ix.0.watchdog_events: 0
SH> dev.ix.0.tso_tx: 174470
SH> dev.ix.0.link_irq: 3
SH> dev.ix.0.queue0.interrupt_rate: 1000000
SH> dev.ix.0.queue0.txd_head: 59
SH> dev.ix.0.queue0.txd_tail: 59
SH> dev.ix.0.queue0.no_desc_avail: 0
SH> dev.ix.0.queue0.tx_packets: 38913
SH> dev.ix.0.queue0.rxd_head: 384
SH> dev.ix.0.queue0.rxd_tail: 383
SH> dev.ix.0.queue0.rx_packets: 54982
SH> dev.ix.0.queue0.rx_bytes: 36197485
SH> dev.ix.0.queue0.lro_queued: 0
SH> dev.ix.0.queue0.lro_flushed: 0
SH> dev.ix.0.queue1.interrupt_rate: 1000000
SH> dev.ix.0.queue1.txd_head: 1417
SH> dev.ix.0.queue1.txd_tail: 1417
SH> dev.ix.0.queue1.no_desc_avail: 0
SH> dev.ix.0.queue1.tx_packets: 51196
SH> dev.ix.0.queue1.rxd_head: 445
SH> dev.ix.0.queue1.rxd_tail: 444
SH> dev.ix.0.queue1.rx_packets: 70841
SH> dev.ix.0.queue1.rx_bytes: 26319740
SH> dev.ix.0.queue1.lro_queued: 0
SH> dev.ix.0.queue1.lro_flushed: 0
SH> dev.ix.0.queue2.interrupt_rate: 20408
SH> dev.ix.0.queue2.txd_head: 194
SH> dev.ix.0.queue2.txd_tail: 194
SH> dev.ix.0.queue2.no_desc_avail: 0
SH> dev.ix.0.queue2.tx_packets: 45102
SH> dev.ix.0.queue2.rxd_head: 696
SH> dev.ix.0.queue2.rxd_tail: 695
SH> dev.ix.0.queue2.rx_packets: 65107
SH> dev.ix.0.queue2.rx_bytes: 49222403
SH> dev.ix.0.queue2.lro_queued: 0
SH> dev.ix.0.queue2.lro_flushed: 0
SH> dev.ix.0.queue3.interrupt_rate: 200000
SH> dev.ix.0.queue3.txd_head: 1605
SH> dev.ix.0.queue3.txd_tail: 1605
SH> dev.ix.0.queue3.no_desc_avail: 0
SH> dev.ix.0.queue3.tx_packets: 77375
SH> dev.ix.0.queue3.rxd_head: 79
SH> dev.ix.0.queue3.rxd_tail: 78
SH> dev.ix.0.queue3.rx_packets: 109498
SH> dev.ix.0.queue3.rx_bytes: 109951775
SH> dev.ix.0.queue3.lro_queued: 0
SH> dev.ix.0.queue3.lro_flushed: 0
SH> dev.ix.0.queue4.interrupt_rate: 10526
SH> dev.ix.0.queue4.txd_head: 1624
SH> dev.ix.0.queue4.txd_tail: 1624
SH> dev.ix.0.queue4.no_desc_avail: 0
SH> dev.ix.0.queue4.tx_packets: 39497
SH> dev.ix.0.queue4.rxd_head: 480
SH> dev.ix.0.queue4.rxd_tail: 479
SH> dev.ix.0.queue4.rx_packets: 51998
SH> dev.ix.0.queue4.rx_bytes: 21965859
SH> dev.ix.0.queue4.lro_queued: 0
SH> dev.ix.0.queue4.lro_flushed: 0
SH> dev.ix.0.queue5.interrupt_rate: 1000000
SH> dev.ix.0.queue5.txd_head: 1613
SH> dev.ix.0.queue5.txd_tail: 1613
SH> dev.ix.0.queue5.no_desc_avail: 0
SH> dev.ix.0.queue5.tx_packets: 69860
SH> dev.ix.0.queue5.rxd_head: 846
SH> dev.ix.0.queue5.rxd_tail: 845
SH> dev.ix.0.queue5.rx_packets: 81331
SH> dev.ix.0.queue5.rx_bytes: 32429926
SH> dev.ix.0.queue5.lro_queued: 0
SH> dev.ix.0.queue5.lro_flushed: 0
SH> dev.ix.0.queue6.interrupt_rate: 142857
SH> dev.ix.0.queue6.txd_head: 1482
SH> dev.ix.0.queue6.txd_tail: 1484
SH> dev.ix.0.queue6.no_desc_avail: 0
SH> dev.ix.0.queue6.tx_packets: 45878
SH> dev.ix.0.queue6.rxd_head: 355
SH> dev.ix.0.queue6.rxd_tail: 354
SH> dev.ix.0.queue6.rx_packets: 62211
SH> dev.ix.0.queue6.rx_bytes: 27653559
SH> dev.ix.0.queue6.lro_queued: 0
SH> dev.ix.0.queue6.lro_flushed: 0
SH> dev.ix.0.queue7.interrupt_rate: 5347
SH> dev.ix.0.queue7.txd_head: 603
SH> dev.ix.0.queue7.txd_tail: 603
SH> dev.ix.0.queue7.no_desc_avail: 0
SH> dev.ix.0.queue7.tx_packets: 61997
SH> dev.ix.0.queue7.rxd_head: 826
SH> dev.ix.0.queue7.rxd_tail: 825
SH> dev.ix.0.queue7.rx_packets: 83460
SH> dev.ix.0.queue7.rx_bytes: 50183116
SH> dev.ix.0.queue7.lro_queued: 0
SH> dev.ix.0.queue7.lro_flushed: 0
SH> dev.ix.0.mac_stats.crc_errs: 0
SH> dev.ix.0.mac_stats.ill_errs: 0
SH> dev.ix.0.mac_stats.byte_errs: 0
SH> dev.ix.0.mac_stats.short_discards: 0
SH> dev.ix.0.mac_stats.local_faults: 3
SH> dev.ix.0.mac_stats.remote_faults: 1
SH> dev.ix.0.mac_stats.rec_len_errs: 0
SH> dev.ix.0.mac_stats.link_xon_txd: 0
SH> dev.ix.0.mac_stats.link_xon_rcvd: 0
SH> dev.ix.0.mac_stats.link_xoff_txd: 0
SH> dev.ix.0.mac_stats.link_xoff_rcvd: 0
SH> dev.ix.0.mac_stats.total_octets_rcvd: 360072702
SH> dev.ix.0.mac_stats.good_octets_rcvd: 359999778
SH> dev.ix.0.mac_stats.total_pkts_rcvd: 637428
SH> dev.ix.0.mac_stats.good_pkts_rcvd: 636321
SH> dev.ix.0.mac_stats.mcast_pkts_rcvd: 35
SH> dev.ix.0.mac_stats.bcast_pkts_rcvd: 1411
SH> dev.ix.0.mac_stats.rx_frames_64: 222251
SH> dev.ix.0.mac_stats.rx_frames_65_127: 159044
SH> dev.ix.0.mac_stats.rx_frames_128_255: 15139
SH> dev.ix.0.mac_stats.rx_frames_256_511: 13885
SH> dev.ix.0.mac_stats.rx_frames_512_1023: 21283
SH> dev.ix.0.mac_stats.rx_frames_1024_1522: 204719
SH> dev.ix.0.mac_stats.recv_undersized: 0
SH> dev.ix.0.mac_stats.recv_fragmented: 0
SH> dev.ix.0.mac_stats.recv_oversized: 0
SH> dev.ix.0.mac_stats.recv_jabberd: 0
SH> dev.ix.0.mac_stats.management_pkts_rcvd: 0
SH> dev.ix.0.mac_stats.management_pkts_drpd: 0
SH> dev.ix.0.mac_stats.checksum_errs: 0
SH> dev.ix.0.mac_stats.good_octets_txd: 882467530
SH> dev.ix.0.mac_stats.total_pkts_txd: 816387
SH> dev.ix.0.mac_stats.good_pkts_txd: 816387
SH> dev.ix.0.mac_stats.bcast_pkts_txd: 36
SH> dev.ix.0.mac_stats.mcast_pkts_txd: 0
SH> dev.ix.0.mac_stats.management_pkts_txd: 0
SH> dev.ix.0.mac_stats.tx_frames_64: 21509
SH> dev.ix.0.mac_stats.tx_frames_65_127: 168051
SH> dev.ix.0.mac_stats.tx_frames_128_255: 19184
SH> dev.ix.0.mac_stats.tx_frames_256_511: 22775
SH> dev.ix.0.mac_stats.tx_frames_512_1023: 24222
SH> dev.ix.0.mac_stats.tx_frames_1024_1522: 560646
SH> dev.ix.0.mac_stats.fc_crc: 0
SH> dev.ix.0.mac_stats.fc_last: 0
SH> dev.ix.0.mac_stats.fc_drpd: 0
SH> dev.ix.0.mac_stats.fc_pkts_rcvd: 0
SH> dev.ix.0.mac_stats.fc_pkts_txd: 0
SH> dev.ix.0.mac_stats.fc_dword_rcvd: 0
SH> dev.ix.0.mac_stats.fc_dword_txd: 0
SH> vmstat -i
SH> interrupt total rate
SH> irq1: atkbd0 1 0
SH> irq6: fdc0 1 0
SH> irq14: ata0 35 0
SH> irq20: uhci0 1 0
SH> irq23: ehci0 41 0
SH> irq66: arcmsr0 66431 74
SH> cpu0: timer 1773313 1999
SH> irq256: ix0:que 0 99591 112
SH> irq257: ix0:que 1 109526 123
SH> irq258: ix0:que 2 97963 110
SH> irq259: ix0:que 3 220346 248
SH> irq260: ix0:que 4 85912 96
SH> irq261: ix0:que 5 155002 174
SH> irq262: ix0:que 6 99027 111
SH> irq263: ix0:que 7 124176 139
SH> irq264: ix0:link 3 0
SH> irq270: igb1:que 0 312 0
SH> irq271: igb1:que 1 2 0
SH> irq274: igb1:link 2 0
SH> cpu7: timer 1765259 1990
SH> cpu6: timer 1765259 1990
SH> cpu4: timer 1765260 1990
SH> cpu5: timer 1765260 1990
SH> cpu1: timer 1765259 1990
SH> cpu2: timer 1765259 1990
SH> cpu3: timer 1765260 1990
SH> Total 15188501 17123
SH> netstat -m
SH> 13479/5091/18570 mbufs in use (current/cache/total)
SH> 12327/4319/16646/524288 mbuf clusters in use (current/cache/total/max)
SH> 12285/1667 mbuf+clusters out of packet secondary zone in use (current/cache)
SH> 6/506/512/262144 4k (page size) jumbo clusters in use (current/cache/total/max)
SH> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
SH> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
SH> 28047K/11934K/39982K bytes allocated to network (current/cache/total)
SH> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
SH> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
SH> 0/0/0 sfbufs in use (current/peak/max)
SH> 0 requests for sfbufs denied
SH> 0 requests for sfbufs delayed
SH> 1257 requests for I/O initiated by sendfile
SH> 0 calls to protocol drain routines
SH> netstat -i
SH> Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
SH> ix0 1500 <Link#1> 00:1b:21:7e:2e:8c 1307940 0 0 1708291 0 0
SH> ix0 1500 X.X.X.0 ixhost 26195 - - 1601162 - -
SH> igb0* 1500 <Link#2> 00:30:48:c5:31:02 0 0 0 0 0 0
SH> igb1 1500 <Link#3> 00:30:48:c5:31:03 741 0 0 721 0 0
SH> igb1 1500 10.10.10.0 10.10.10.64 679 - - 718 - -
SH> lo0 16384 <Link#4> 6824 0 0 6824 0 0
SH> lo0 16384 fe80:4::1 fe80:4::1 0 - - 0 - -
SH> lo0 16384 localhost ::1 0 - - 0 - -
SH> lo0 16384 your-net localhost 26 - - 6824 - -
SH> A ping from the cisco 6509 its connected to:-
SH> Sending 1000, 100-byte ICMP Echos to 85.236.96.64, timeout is 2 seconds:
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
SH> !!!!!!!!!!!!!!!!!!!!
SH> Success rate is 99 percent (996/1000), round-trip min/avg/max = 1/1/208 ms
SH> Config on the cisco end:-
SH> TenGigabitEthernet9/2 is up, line protocol is up (connected)
SH> Hardware is C6k 10000Mb 802.3, address is 001e.1323.f325 (bia 001e.1323.f325)
SH> Description: ixhost (10Gbps)
SH> MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
SH> reliability 255/255, txload 1/255, rxload 1/255
SH> Encapsulation ARPA, loopback not set
SH> Keepalive set (10 sec)
SH> Full-duplex, 10Gb/s
SH> input flow-control is on, output flow-control is on
SH> ARP type: ARPA, ARP Timeout 04:00:00
SH> Last input never, output 46w5d, output hang never
SH> Last clearing of "show interface" counters never
SH> Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
SH> Queueing strategy: fifo
SH> Output queue: 0/40 (size/max)
SH> 5 minute input rate 12204000 bits/sec, 1351 packets/sec
SH> 5 minute output rate 4998000 bits/sec, 1007 packets/sec
SH> 78180252111 packets input, 92996518599740 bytes, 0 no buffer
SH> Received 314449 broadcasts (0 multicasts)
SH> 0 runts, 0 giants, 0 throttles
SH> 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
SH> 0 watchdog, 0 multicast, 0 pause input
SH> 0 input packets with dribble condition detected
SH> 51832915763 packets output, 24954526878125 bytes, 0 underruns
SH> 0 output errors, 0 collisions, 3 interface resets
SH> 0 babbles, 0 late collision, 0 deferred
SH> 0 lost carrier, 0 no carrier, 0 PAUSE output
SH> 0 output buffer failures, 0 output buffers swapped out
SH> systat 1 :if
SH> /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
SH> Load Average |
SH> Interface Traffic Peak Total
SH> lo0 in 0.953 KB/s 1.916 KB/s 2.669 MB
SH> out 0.953 KB/s 1.916 KB/s 2.669 MB
SH> igb1 in 0.063 KB/s 0.128 KB/s 85.663 KB
SH> out 0.142 KB/s 0.269 KB/s 203.177 KB
SH> ix0 in 215.019 KB/s 679.274 KB/s 1000.871 MB
SH> out 755.770 KB/s 1.113 MB/s 2.447 GB
SH> Regards
SH> Steve
can you show netstat -Q ?
--
С уважением,
Коньков mailto:kes-kes at yandex.ru
More information about the freebsd-net
mailing list