Problem on re0

[YongHyeon PYUN]

On Sat, Dec 17, 2011 at 06:49:48PM +0200, Sami Halabi wrote:
> Hi,
> 
> I've moved to a new server with bge card driver, similar configuration:
> /etc/sysctl.conf
> net.inet.flowtable.enable=0
> net.inet.ip.fastforwarding=1
> kern.ipc.somaxconn=8192
> kern.ipc.shmmax=2147483648
> kern.ipc.maxsockets=204800
> kern.ipc.maxsockbuf=2097152
> hw.intr_storm_threshold=9000
> kern.maxfiles=256000
> kern.maxfilesperproc=230400
> net.inet.ip.dummynet.pipe_slot_limit=1000
> #net.inet.ip.dummynet.io_fast=1
> net.link.ether.ipfw=1
> kern.ipc.nmbclusters=409600
> net.graph.recvspace=40960
> net.graph.maxdgram=40960
> 
>  Kernel
> -----------
> device          lagg
> options         IPFIREWALL
> options         IPFIREWALL_FORWARD
> options         IPFIREWALL_VERBOSE
> options         DUMMYNET
> options         HZ=1000
> options         TCP_SIGNATURE
> device          crypto          # core crypto support
> device          cryptodev       # /dev/crypto for access to h/w
> options         IPSEC
> options         DEVICE_POLLING
> device if_bridge
> 
> /boot/loader.conf
> net.graph.maxalloc=128000
> net.graph.maxdata=128000
> net.graph.threads=4
> 
> and i have similar problem.. i check each minute for ping and i see
> sometimes loss of 50% or 100% (ping -c 2 -t 4 otherside.ip.com),
> here are the sysctl dev.bge.1.stats:
> dev.bge.1.stats.FramesDroppedDueToFilters: 0
> dev.bge.1.stats.DmaWriteQueueFull: 2291350
> dev.bge.1.stats.DmaWriteHighPriQueueFull: 0
> dev.bge.1.stats.NoMoreRxBDs: 0
> dev.bge.1.stats.InputDiscards: 0
> dev.bge.1.stats.InputErrors: 0
> dev.bge.1.stats.RecvThresholdHit: 124120300
> dev.bge.1.stats.DmaReadQueueFull: 13470948
> dev.bge.1.stats.DmaReadHighPriQueueFull: 962
> dev.bge.1.stats.SendDataCompQueueFull: 0
> dev.bge.1.stats.RingSetSendProdIndex: 349289174
> dev.bge.1.stats.RingStatusUpdate: 260304688
> dev.bge.1.stats.Interrupts: 260304688
> dev.bge.1.stats.AvoidedInterrupts: 0
> dev.bge.1.stats.SendThresholdHit: 0
> dev.bge.1.stats.rx.ifHCInOctets: 2165908673
> dev.bge.1.stats.rx.Fragments: 0
> dev.bge.1.stats.rx.UnicastPkts: 237503495
> dev.bge.1.stats.rx.MulticastPkts: 0
> dev.bge.1.stats.rx.FCSErrors: 0
> dev.bge.1.stats.rx.AlignmentErrors: 0
> dev.bge.1.stats.rx.xonPauseFramesReceived: 0
> dev.bge.1.stats.rx.xoffPauseFramesReceived: 0
> dev.bge.1.stats.rx.ControlFramesReceived: 0
> dev.bge.1.stats.rx.xoffStateEntered: 0
> dev.bge.1.stats.rx.FramesTooLong: 0
> dev.bge.1.stats.rx.Jabbers: 0
> dev.bge.1.stats.rx.UndersizePkts: 0
> dev.bge.1.stats.rx.inRangeLengthError: 0
> dev.bge.1.stats.rx.outRangeLengthError: 0
> dev.bge.1.stats.tx.ifHCOutOctets: 1956421618
> dev.bge.1.stats.tx.Collisions: 0
> dev.bge.1.stats.tx.XonSent: 0
> dev.bge.1.stats.tx.XoffSent: 0
> dev.bge.1.stats.tx.flowControlDone: 0
> dev.bge.1.stats.tx.InternalMacTransmitErrors: 0
> dev.bge.1.stats.tx.SingleCollisionFrames: 0
> dev.bge.1.stats.tx.MultipleCollisionFrames: 0
> dev.bge.1.stats.tx.DeferredTransmissions: 0
> dev.bge.1.stats.tx.ExcessiveCollisions: 0
> dev.bge.1.stats.tx.LateCollisions: 0
> dev.bge.1.stats.tx.UnicastPkts: 347260508
> dev.bge.1.stats.tx.MulticastPkts: 0
> dev.bge.1.stats.tx.BroadcastPkts: 30306
> dev.bge.1.stats.tx.CarrierSenseErrors: 0
> dev.bge.1.stats.tx.Discards: 0
> dev.bge.1.stats.tx.Errors: 0
> 
> this driver gives more statics...
> 
> please help me, this causes us serious problems with customers.
> 

Given that you see the same issue with bge(4) it looks like the
root cause is not in ethernet driver.  I also see no evidence of
dropped frames from bge(4) hardware MAC statistics.
It seems you have really complex network configuration(ipfw(4),
lagg(4), dummynet(4), bridge(4) and ipsec(4) etc).  Finding out
simplest network setup that shows the issue would be required here.