bce packet loss

[Charles Sprickman]

Hello,

We're running a few 8.1-R servers with Broadcom bce interfaces (Dell R510) 
and I'm seeing occasional packet loss on them (enough that it trips nagios 
now and then).  Cabling seems fine as neither the switch nor the sysctl 
info for the device show any errors/collisions/etc, however there is one 
odd one, which is "dev.bce.1.stat_IfHCInBadOctets: 539369".  See [1] below 
for full sysctl output.  The switch shows no errors but for "Dropped 
packets 683868".

pciconf output is also below. [2]

By default, the switch had flow control set to "on".  I also let it run 
with "auto".  In both cases, the drops continued to increment.  I'm now 
running with flow control off to see if that changes anything.

I do see some correlation between cpu usage and drops - I have cpu usage 
graphed in nagios and cacti is graphing the drops on the dell switch. 
There's no signs of running out of mbufs or similar.

So given that limited info, is there anything I can look at to track this 
down?  Anything stand out in the stats sysctl exposes?  Two things are 
standing out for me - the number of changes in bce regarding flow control 
that are not in 8.1, and the correlation between cpu load and the drops.

What other information can I provide?

Thanks,

Charles

[1] [root at h23 /home/spork]# sysctl -a |grep bce.1
dev.bce.1.%desc: Broadcom NetXtreme II BCM5716 1000Base-T (C0)
dev.bce.1.%driver: bce
dev.bce.1.%location: slot=0 function=1
dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x163b subvendor=0x1028 
subdevice=0x02f1 class=0x020000
dev.bce.1.%parent: pci1
dev.bce.1.l2fhdr_error_count: 0
dev.bce.1.mbuf_alloc_failed_count: 282
dev.bce.1.fragmented_mbuf_count: 2748
dev.bce.1.dma_map_addr_rx_failed_count: 0
dev.bce.1.dma_map_addr_tx_failed_count: 5
dev.bce.1.unexpected_attention_count: 0
dev.bce.1.stat_IfHcInOctets: 62708651108
dev.bce.1.stat_IfHCInBadOctets: 539369
dev.bce.1.stat_IfHCOutOctets: 434264587173
dev.bce.1.stat_IfHCOutBadOctets: 0
dev.bce.1.stat_IfHCInUcastPkts: 533441918
dev.bce.1.stat_IfHCInMulticastPkts: 3108746
dev.bce.1.stat_IfHCInBroadcastPkts: 1314905
dev.bce.1.stat_IfHCOutUcastPkts: 640961970
dev.bce.1.stat_IfHCOutMulticastPkts: 26
dev.bce.1.stat_IfHCOutBroadcastPkts: 8909
dev.bce.1.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.1.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.1.stat_Dot3StatsFCSErrors: 0
dev.bce.1.stat_Dot3StatsAlignmentErrors: 0
dev.bce.1.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.1.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.1.stat_Dot3StatsLateCollisions: 0
dev.bce.1.stat_EtherStatsCollisions: 0
dev.bce.1.stat_EtherStatsFragments: 0
dev.bce.1.stat_EtherStatsJabbers: 0
dev.bce.1.stat_EtherStatsUndersizePkts: 0
dev.bce.1.stat_EtherStatsOversizePkts: 0
dev.bce.1.stat_EtherStatsPktsRx64Octets: 34048797
dev.bce.1.stat_EtherStatsPktsRx65Octetsto127Octets: 431844366
dev.bce.1.stat_EtherStatsPktsRx128Octetsto255Octets: 25946173
dev.bce.1.stat_EtherStatsPktsRx256Octetsto511Octets: 39936369
dev.bce.1.stat_EtherStatsPktsRx512Octetsto1023Octets: 2296565
dev.bce.1.stat_EtherStatsPktsRx1024Octetsto1522Octets: 3931392
dev.bce.1.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.1.stat_EtherStatsPktsTx64Octets: 60122571
dev.bce.1.stat_EtherStatsPktsTx65Octetsto127Octets: 221041349
dev.bce.1.stat_EtherStatsPktsTx128Octetsto255Octets: 40177071
dev.bce.1.stat_EtherStatsPktsTx256Octetsto511Octets: 24099944
dev.bce.1.stat_EtherStatsPktsTx512Octetsto1023Octets: 44493532
dev.bce.1.stat_EtherStatsPktsTx1024Octetsto1522Octets: 251036438
dev.bce.1.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.1.stat_XonPauseFramesReceived: 61778
dev.bce.1.stat_XoffPauseFramesReceived: 76315
dev.bce.1.stat_OutXonSent: 0
dev.bce.1.stat_OutXoffSent: 0
dev.bce.1.stat_FlowControlDone: 0
dev.bce.1.stat_MacControlFramesReceived: 0
dev.bce.1.stat_XoffStateEntered: 0
dev.bce.1.stat_IfInFramesL2FilterDiscards: 145832
dev.bce.1.stat_IfInRuleCheckerDiscards: 0
dev.bce.1.stat_IfInFTQDiscards: 0
dev.bce.1.stat_IfInMBUFDiscards: 0
dev.bce.1.stat_IfInRuleCheckerP4Hit: 4448215
dev.bce.1.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.1.stat_CatchupInFTQDiscards: 0
dev.bce.1.stat_CatchupInMBUFDiscards: 0
dev.bce.1.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.1.com_no_buffers: 0

[2] pciconf -lvb
bce1 at pci0:1:0:1:	class=0x020000 card=0x02f11028 chip=0x163b14e4 
rev=0x20 hdr=0x00
     vendor     = 'Broadcom Corporation'
     class      = network
     subclass   = ethernet
     bar   [10] = type Memory, range 64, base 0xdc000000, size 33554432, 
enabled