FreeBSD 10 network flapping, ix driver unreliable?

[George Neville-Neil]

On Feb 15, 2014, at 21:32 , Kevin Bowling <kevin.bowling at kev009.com> wrote:

> On 2/15/2014 4:43 PM, George Neville-Neil wrote:
>> 
>> On Feb 15, 2014, at 15:14 , Kevin Bowling <kevin.bowling at kev009.com> wrote:
>> 
>>> Hi,
>>> 
>>> I have FreeBSD 10.0-RELEASE installed on two Dell C6100 nodes.  Each node has an Intel X520-DA2 dual port 10gig card.  One of the ports on each go to a switch using direct attach coaxial cables.  The other port is directly connected between the two nodes (think crossover in twisted pair terminology) again using direct attach coaxial cables.
>>> 
>>> On both machines, and on both ports (including the "crossover"), the links flap several times per day.
>>> 
>>> I've pasted the output of lspci -vv and dmesg here:
>>> https://gist.github.com/kev009/9024442
>>> 
>>> There's nothing outstanding about the setup otherwise.  I suspected some interaction with the switch initially but the "crossover" has eliminated that suspicion.
>>> 
>>> It seems the ix driver is not very reliable under common conditions, i.e. https://forums.freebsd.org/viewtopic.php?f=7&t=44570 and a search of this list.  Any recommendations or tests?
>>> 
>> 
>> Can you post (to your gist link) the output of sysctl dev.ix ?
> 
> Hi George,
> 
> sysctl info added to gist link.  ix0 has been up for around 27 days. ix1 for about 24hrs.
> 

I think this has something to do with it.

dev.ix.0.mac_stats.local_faults: 314
dev.ix.0.mac_stats.remote_faults: 41

The device is seeing errors at the MAC layer, which  I don’t think a driver bug would
cause, though there is always the possibility of a misconfiguration causing flapping.
Can you try different cables?

When you hook it to the switch does the switch give better diagnostics?  Reading
over the Intel 82599 chip manual is not, shall we say, illuminating, 
"Number of faults in the local MAC. This register is valid only when the link speed is 10 Gb/s.”

Best,
George