igb driver tx hangs when out of mbuf clusters

[Karim Fodil-Lemelin]

Hi List,

The issue is with the igb driver from 7.4 RC3 r218406. If the driver runs
out of mbuf clusters it simply stops receiving even after the clusters have
been freed.

igb0 at pci0:7:0:0:        class=0x020000 card=0x00008086 chip=0x10a78086
rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = network
    subclass   = ethernet

Here is how to reproduce:

1) Set mbuf cluster count very low (4096)
2) do an intensive UDP or TCP transfer

Some pointers:

- After few seconds or minutes maximum, the traffic does not flow any more
across the machine.
- At this point we see the InErrors jump in a burst on network iterface
igb0. Then, all packets in are counted as In Errors.
- Their number is exactly equal to dev.igb.0.mac_stats.missed_packets, and
not far from dev.igb.0.mac_stats.xoff_txd (Note that one is on receive the
other on send).
- Also we see this counter goes non-null: dev.igb.0.mac_stats.recv_no_buff
- We tried to enable/disable TSO, Checksum on receive and send, flow control
and aim. The problem still shows up, only the time to get there is
different.
- Forcing the driver to use one queue (static int igb_num_queues = 1;) did
not help.
- Using igb_header_split=TRUE does help a bit but not unlike disabling
hardware acceleration it just takes more time (probably because its using
mbufs more instead of clusters and there is 4times more mbufs then
clusters).
- Using larger values for mbufs cluster also delays the problem

We think the code in "igb_refresh_mbufs()" does not handle well running out
of mbufs but that is how far we can get at the moment. We are seeking your
help and wisdom, and are willing to test patches or suggested settings.

Thanks in advance,

Karim.