suboptimal bge(4) BCM5704 performance in RELENG_8

[YongHyeon PYUN]

On Wed, Mar 07, 2012 at 03:12:32PM +0700, Eugene Grosbein wrote:
> 08.03.2012 03:29, YongHyeon PYUN пишет:
> > On Tue, Mar 06, 2012 at 07:54:02PM +0700, Eugene Grosbein wrote:
> >> Hi!
> >>
> >> Yesterday I've updated old HP ProLiant DL360 G4p to 8.3-PRELELEASE/amd64
> >> running busy icecast2 server in hope it can saturate 1G bge(4) link.
> >>
> >> This server has PCI-X connected HP NC7782 Gigabit Server Adapter (BCM5704).
> > 
> > Would you show me the output of dmesg(bge(4) and brgphy(4) related
> > ones)?
> 
> # egrep 'bge|brgphy' /var/run/dmesg.boot 
> bge0: <HP NC7782 Gigabit Server Adapter, ASIC rev. 0x002100> mem 0xfdf70000-0xfdf7ffff irq 25 at device 2.0 on pci2
> bge0: CHIP ID 0x00002100; ASIC REV 0x02; CHIP REV 0x21; PCI-X
> miibus0: <MII bus> on bge0
> brgphy0: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus0
> brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
> bge0: Ethernet address: 00:18:fe:86:ab:f4
> bge0: [ITHREAD]
> bge1: <HP NC7782 Gigabit Server Adapter, ASIC rev. 0x002100> mem 0xfdf60000-0xfdf6ffff irq 26 at device 2.1 on pci2
> bge1: CHIP ID 0x00002100; ASIC REV 0x02; CHIP REV 0x21; PCI-X
> miibus1: <MII bus> on bge1
> brgphy1: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus1
> brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
> bge1: Ethernet address: 00:18:fe:86:ab:f3
> bge1: [ITHREAD]
> 

It seems your controller is BCM5704 B2 and it is running in PCI-X
mode.  There is one known BCM5704 issue but I don't think you're
seeing this one.  To confirm that could you show me both the output
of "pciconf -lcbv" and "devinfo -rv"?


> There are bge0 and bge1 but bge1 is not used and is down.
> 
> >> Is it supposed to emit more than 540Mbit/s with average packet size equal to 1300?
> >>
> > 
> > Yes.
> 
> [skip]
> 
> >> I've glanced at bge(4) code and see it uses lots of hardcoded constants
> >> for rx/tx descriptor rings, for interrupt moderation and for interface FIFO queue.
> > 
> > Firmware has a fixed number of descriptors so increasing them
> > wouldn't give better numbers.
> 
> What it the limit?
> 

Both TX and RX have 512 descriptors.

> >> And no loader tunnables/sysctls like em/igb have.
> > 
> > Show me the output of "sysctl dev.bge.0.stats".
> 
> # sysctl dev.bge.0.stats
> dev.bge.0.stats.FramesDroppedDueToFilters: 0
> dev.bge.0.stats.DmaWriteQueueFull: 84072
> dev.bge.0.stats.DmaWriteHighPriQueueFull: 0
> dev.bge.0.stats.NoMoreRxBDs: 0
> dev.bge.0.stats.InputDiscards: 0
> dev.bge.0.stats.InputErrors: 30
> dev.bge.0.stats.RecvThresholdHit: 745400662
> dev.bge.0.stats.DmaReadQueueFull: 2020586592
> dev.bge.0.stats.DmaReadHighPriQueueFull: 0
> dev.bge.0.stats.SendDataCompQueueFull: 0
> dev.bge.0.stats.RingSetSendProdIndex: 2832885493
> dev.bge.0.stats.RingStatusUpdate: 899990835
> dev.bge.0.stats.Interrupts: 899990835
> dev.bge.0.stats.AvoidedInterrupts: 0
> dev.bge.0.stats.SendThresholdHit: 0
> dev.bge.0.stats.rx.ifHCInOctets: 491268800
> dev.bge.0.stats.rx.Fragments: 234
> dev.bge.0.stats.rx.UnicastPkts: 1977202324
> dev.bge.0.stats.rx.MulticastPkts: 0
> dev.bge.0.stats.rx.FCSErrors: 341
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You have multiple FCS and Input errors. Check signal
quality(i.e. UTP cable).

> dev.bge.0.stats.rx.AlignmentErrors: 0
> dev.bge.0.stats.rx.xonPauseFramesReceived: 0
> dev.bge.0.stats.rx.xoffPauseFramesReceived: 0
> dev.bge.0.stats.rx.ControlFramesReceived: 0
> dev.bge.0.stats.rx.xoffStateEntered: 0
> dev.bge.0.stats.rx.FramesTooLong: 0
> dev.bge.0.stats.rx.Jabbers: 0
> dev.bge.0.stats.rx.UndersizePkts: 0
> dev.bge.0.stats.rx.inRangeLengthError: 0
> dev.bge.0.stats.rx.outRangeLengthError: 0
> dev.bge.0.stats.tx.ifHCOutOctets: 683592718
> dev.bge.0.stats.tx.Collisions: 0
> dev.bge.0.stats.tx.XonSent: 0
> dev.bge.0.stats.tx.XoffSent: 0
> dev.bge.0.stats.tx.flowControlDone: 0
> dev.bge.0.stats.tx.InternalMacTransmitErrors: 0
> dev.bge.0.stats.tx.SingleCollisionFrames: 0
> dev.bge.0.stats.tx.MultipleCollisionFrames: 0
> dev.bge.0.stats.tx.DeferredTransmissions: 0
> dev.bge.0.stats.tx.ExcessiveCollisions: 0
> dev.bge.0.stats.tx.LateCollisions: 0
> dev.bge.0.stats.tx.UnicastPkts: 3292778353
> dev.bge.0.stats.tx.MulticastPkts: 0
> dev.bge.0.stats.tx.BroadcastPkts: 147
> dev.bge.0.stats.tx.CarrierSenseErrors: 0
> dev.bge.0.stats.tx.Discards: 0
> dev.bge.0.stats.tx.Errors: 0
> 
> >> Should I try to play with constants in the code and if so, what are limits of this chip?
> >> Or it will never be capable of utilizing full gigabit speed?
> > 
> > BCM5704 is old controller but it should have no problems to
> > saturate the link.  I suspect it could be related with DMA
> > configuration but needs more information.
> 
> I'll supply any information. It's also possible to try kernel patches, if any :-)

Does netperf benchmark also show 540Mbps on bge(4)?

> 
> Eugene Grosbein