FreeBSD 9.3 - Intel X520-SR2 stops passing packets

Fri May 22 16:48:14 UTC 2015

> On May 22, 2015, at 10:21 AM, Guy Helmer <guy.helmer at gmail.com> wrote:
> 
> 
>> On May 21, 2015, at 6:09 PM, Eric Joyner <erj at freebsd.org <mailto:erj at freebsd.org>> wrote:
>> 
>> Are there any log messages printed out by the driver? The sysctls don't really look out of the ordinary, other than the number of sub-64 byte packets.
> 
> Not that I could tell — grep of /var/log/messages and the text from the rotated messages.*.gz logs yielded nothing.
> 
> Guy

This is interesting - I was not looking at the Ierrs or Idrop columns of netstat before, but as the problem has recurred, I’ve found it’s dropping nearly all packets received on the ix interfaces:

> netstat -i
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ix0    1500 <Link#3>      90:e2:ba:35:aa:8c 8310484539  3913 16255003933        0     0     0
ix0    1500 fe80::92e2:ba fe80::92e2:baff:f        0     -     -        0     -     -
ix1    1500 <Link#4>      90:e2:ba:35:aa:8d 6897310372 58387379 8207969523        0     0     0
ix1    1500 fe80::92e2:ba fe80::92e2:baff:f        0     -     -        0     -     -

> netstat -i
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ix0    1500 <Link#3>      90:e2:ba:35:aa:8c 8310484538  3920 16299121753        0     0     0
ix0    1500 fe80::92e2:ba fe80::92e2:baff:f        0     -     -        0     -     -
ix1    1500 <Link#4>      90:e2:ba:35:aa:8d 6897310369 58470675 8237532419        0     0     0
ix1    1500 fe80::92e2:ba fe80::92e2:baff:f        0     -     -        0     -     -

# vmstat -z  — related to mbufs:
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
mbuf_packet:            256, 12982305,   38834,    5203,9171090979,196374,   0
mbuf:                   256, 12982305,       3,  167430,8518885166,   0,   0
mbuf_cluster:          2048, 262144,   44037,    9849,70443362,399118,892109
mbuf_jumbo_page:       4096, 1014242,       0,    4619, 9721759,   0,   0
mbuf_jumbo_9k:         9216, 300516,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 169040,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0

# vmstat -z | grep mbuf
mbuf_packet:            256, 12982305,   38834,    5203,9171103748,196374,   0
mbuf:                   256, 12982305,       5,  167428,8518902143,   0,   0
mbuf_cluster:          2048, 262144,   44037,    9849,70443362,399118,892109
mbuf_jumbo_page:       4096, 1014242,       0,    4747, 9721896,   0,   0
mbuf_jumbo_9k:         9216, 300516,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 169040,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0

No interface errors logging in /var/log/messages…

ifconfig down / ifconfig up on the ix0 and ix1 interfaces gets traffic flowing again.

Guy

> 
>> 
>> On Thu, May 21, 2015 at 9:10 AM Guy Helmer <guy.helmer at gmail.com <mailto:guy.helmer at gmail.com>> wrote:
>> 
>> > On May 21, 2015, at 8:52 AM, Christopher Forgeron <csforgeron at gmail.com <mailto:csforgeron at gmail.com>> wrote:
>> >
>> > A few things:
>> >
>> > 1) How long before you have this behaviour?
>> >
>> > 2) What's the output of 'netstat -m' when you have the problem?
>> >
>> > 3) What is your MTU set to, and do you have TSO on or off?
>> >
>> > On Thu, May 21, 2015 at 10:33 AM, Guy Helmer <guy.helmer at gmail.com <mailto:guy.helmer at gmail.com> <mailto:guy.helmer at gmail.com <mailto:guy.helmer at gmail.com>>> wrote:
>> > I’ve noticed that there have been reports of problems with Intel X520-SR2 network interfaces stopping working. I think I’m seeing a similar issue where the 10Gb interfaces stop receiving traffic (they’re being used in promiscuous mode to sniff traffic from a tap). ifconfig shows the interfaces are still active and the links are OK. ifconfig down/up restores activity. I’ve changed hw.intr_storm_threshold=8000 but I couldn’t tell if the interrupt storm threshold had been triggered at the time the interfaces stopped passing traffic.
>> 
>> It seems to run from hours to days without problems.
>> 
>> I don’t have the output of “netstat -m” available, but it did not indicate any mbuf or cluster allocation failures. No jumbo clusters (4k, 9k, or 16k) were allocated.
>> 
>> MTU is 1500. TSO is “on” but would seem to be irrelevant — no packets are transmitted out of these interfaces (verified using “netstat -i”).
>> 
>> Thanks,
>> Guy
>> _______________________________________________
>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net <http://lists.freebsd.org/mailman/listinfo/freebsd-net>
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org <mailto:freebsd-net-unsubscribe at freebsd.org>"
> 