Fwd: igb driver tx hangs when out of mbuf clusters

[Karim Fodil-Lemelin]

Subject: Re: igb driver tx hangs when out of mbuf clusters

> To: Lev Serebryakov <lev at serebryakov.spb.ru>
> Cc: freebsd-net at freebsd.org
>
>
> 2011/2/7 Lev Serebryakov <lev at serebryakov.spb.ru>
>
> Hello, Karim.
>> You wrote 7 февраля 2011 г., 19:58:04:
>>
>>
>> > The issue is with the igb driver from 7.4 RC3 r218406. If the driver
>> runs
>> > out of mbuf clusters it simply stops receiving even after the clusters
>> have
>> > been freed.
>>   It looks like my problems with em0 (see thread "em0 hangs without
>>  any messages like "Watchdog timeout", only down/up reset it.")...
>>  Codebase for em and igb is somewhat common...
>>
>> --
>> // Black Lion AKA Lev Serebryakov <lev at serebryakov.spb.ru>
>>
>> I agree.
>
> Do you get missed packets in mac_stats (sysctl dev.em | grep missed)?
>
> I might not have mentioned but I can also 'fix' the problem by doing
> ifconfig igb0 down/up.
>
> I will try using POLLING to 'automatize' the reset as you mentioned in your
> thread.
>
> Karim.
>
>
Follow up on tests with POLLING: The problem is still occurring although it
takes more time ... Outputs of sysctl dev.igb0 and netstat -m will follow:

9219/99426/108645 mbufs in use (current/cache/total)
9217/90783/100000/100000 mbuf clusters in use (current/cache/total/max)
0/640 mbuf+clusters out of packet secondary zone in use (current/cache)
0/12800/12800/12800 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
20738K/257622K/278361K bytes allocated to network (current/cache/total)
0/291/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/5/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7
dev.igb.0.%driver: igb
dev.igb.0.%location: slot=0 function=0
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086
subdevice=0x0000 class=0x020000
dev.igb.0.%parent: pci7
dev.igb.0.nvm: -1
dev.igb.0.flow_control: 3
dev.igb.0.enable_aim: 1
dev.igb.0.rx_processing_limit: 100
dev.igb.0.link_irq: 4
dev.igb.0.dropped: 0
dev.igb.0.tx_dma_fail: 0
dev.igb.0.rx_overruns: 464
dev.igb.0.watchdog_timeouts: 0
dev.igb.0.device_control: 1490027073
dev.igb.0.rx_control: 67141658
dev.igb.0.interrupt_mask: 0
dev.igb.0.extended_int_mask: 0
dev.igb.0.tx_buf_alloc: 14
dev.igb.0.rx_buf_alloc: 34
dev.igb.0.fc_high_water: 29488
dev.igb.0.fc_low_water: 29480
dev.igb.0.queue0.interrupt_rate: 111111
dev.igb.0.queue0.txd_head: 877
dev.igb.0.queue0.txd_tail: 877
dev.igb.0.queue0.no_desc_avail: 0
dev.igb.0.queue0.tx_packets: 92013
dev.igb.0.queue0.rxd_head: 570
dev.igb.0.queue0.rxd_tail: 570
dev.igb.0.queue0.rx_packets: 163386
dev.igb.0.queue0.rx_bytes: 240260310
dev.igb.0.queue0.lro_queued: 0
dev.igb.0.queue0.lro_flushed: 0
dev.igb.0.mac_stats.excess_coll: 0
dev.igb.0.mac_stats.single_coll: 0
dev.igb.0.mac_stats.multiple_coll: 0
dev.igb.0.mac_stats.late_coll: 0
dev.igb.0.mac_stats.collision_count: 0
dev.igb.0.mac_stats.symbol_errors: 0
dev.igb.0.mac_stats.sequence_errors: 0
dev.igb.0.mac_stats.defer_count: 0
dev.igb.0.mac_stats.missed_packets: 3104
dev.igb.0.mac_stats.recv_no_buff: 4016
dev.igb.0.mac_stats.recv_undersize: 0
dev.igb.0.mac_stats.recv_fragmented: 0
dev.igb.0.mac_stats.recv_oversize: 0
dev.igb.0.mac_stats.recv_jabber: 0
dev.igb.0.mac_stats.recv_errs: 0
dev.igb.0.mac_stats.crc_errs: 0
dev.igb.0.mac_stats.alignment_errs: 0
dev.igb.0.mac_stats.coll_ext_errs: 0
dev.igb.0.mac_stats.xon_recvd: 0
dev.igb.0.mac_stats.xon_txd: 346
dev.igb.0.mac_stats.xoff_recvd: 0
dev.igb.0.mac_stats.xoff_txd: 3450
dev.igb.0.mac_stats.total_pkts_recvd: 166515
dev.igb.0.mac_stats.good_pkts_recvd: 163411
dev.igb.0.mac_stats.bcast_pkts_recvd: 0
dev.igb.0.mac_stats.mcast_pkts_recvd: 51
dev.igb.0.mac_stats.rx_frames_64: 10
dev.igb.0.mac_stats.rx_frames_65_127: 1601
dev.igb.0.mac_stats.rx_frames_128_255: 53
dev.igb.0.mac_stats.rx_frames_256_511: 42
dev.igb.0.mac_stats.rx_frames_512_1023: 18
dev.igb.0.mac_stats.rx_frames_1024_1522: 161687
dev.igb.0.mac_stats.good_octets_recvd: 240948229
dev.igb.0.mac_stats.good_octets_txd: 5947150
dev.igb.0.mac_stats.total_pkts_txd: 95809
dev.igb.0.mac_stats.good_pkts_txd: 92013
dev.igb.0.mac_stats.bcast_pkts_txd: 1516
dev.igb.0.mac_stats.mcast_pkts_txd: 1817
dev.igb.0.mac_stats.tx_frames_64: 90302
dev.igb.0.mac_stats.tx_frames_65_127: 1711
dev.igb.0.mac_stats.tx_frames_128_255: 0
dev.igb.0.mac_stats.tx_frames_256_511: 0
dev.igb.0.mac_stats.tx_frames_512_1023: 0
dev.igb.0.mac_stats.tx_frames_1024_1522: 0
dev.igb.0.mac_stats.tso_txd: 0
dev.igb.0.mac_stats.tso_ctx_fail: 0
dev.igb.0.interrupts.asserts: 5584
dev.igb.0.interrupts.rx_pkt_timer: 163411
dev.igb.0.interrupts.rx_abs_timer: 163386
dev.igb.0.interrupts.tx_pkt_timer: 92013
dev.igb.0.interrupts.tx_abs_timer: 0
dev.igb.0.interrupts.tx_queue_empty: 92013
dev.igb.0.interrupts.tx_queue_min_thresh: 0
dev.igb.0.interrupts.rx_desc_min_thresh: 19
dev.igb.0.interrupts.rx_overrun: 0
dev.igb.0.host.breaker_tx_pkt: 0
dev.igb.0.host.host_tx_pkt_discard: 0
dev.igb.0.host.rx_pkt: 0
dev.igb.0.host.breaker_rx_pkts: 0
dev.igb.0.host.breaker_rx_pkt_drop: 0
dev.igb.0.host.tx_good_pkt: 0
dev.igb.0.host.breaker_tx_pkt_drop: 0
dev.igb.0.host.rx_good_bytes: 240948229
dev.igb.0.host.tx_good_bytes: 5947150
dev.igb.0.host.length_errors: 0
dev.igb.0.host.serdes_violation_pkt: 0
dev.igb.0.host.header_redir_missed: 0