suboptimal bge(4) BCM5704 performance in RELENG_8

[Eugene Grosbein]

08.03.2012 03:29, YongHyeon PYUN пишет:
> On Tue, Mar 06, 2012 at 07:54:02PM +0700, Eugene Grosbein wrote:
>> Hi!
>>
>> Yesterday I've updated old HP ProLiant DL360 G4p to 8.3-PRELELEASE/amd64
>> running busy icecast2 server in hope it can saturate 1G bge(4) link.
>>
>> This server has PCI-X connected HP NC7782 Gigabit Server Adapter (BCM5704).
> 
> Would you show me the output of dmesg(bge(4) and brgphy(4) related
> ones)?

# egrep 'bge|brgphy' /var/run/dmesg.boot 
bge0: <HP NC7782 Gigabit Server Adapter, ASIC rev. 0x002100> mem 0xfdf70000-0xfdf7ffff irq 25 at device 2.0 on pci2
bge0: CHIP ID 0x00002100; ASIC REV 0x02; CHIP REV 0x21; PCI-X
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:18:fe:86:ab:f4
bge0: [ITHREAD]
bge1: <HP NC7782 Gigabit Server Adapter, ASIC rev. 0x002100> mem 0xfdf60000-0xfdf6ffff irq 26 at device 2.1 on pci2
bge1: CHIP ID 0x00002100; ASIC REV 0x02; CHIP REV 0x21; PCI-X
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge1: Ethernet address: 00:18:fe:86:ab:f3
bge1: [ITHREAD]

There are bge0 and bge1 but bge1 is not used and is down.

>> Is it supposed to emit more than 540Mbit/s with average packet size equal to 1300?
>>
> 
> Yes.

[skip]

>> I've glanced at bge(4) code and see it uses lots of hardcoded constants
>> for rx/tx descriptor rings, for interrupt moderation and for interface FIFO queue.
> 
> Firmware has a fixed number of descriptors so increasing them
> wouldn't give better numbers.

What it the limit?

>> And no loader tunnables/sysctls like em/igb have.
> 
> Show me the output of "sysctl dev.bge.0.stats".

# sysctl dev.bge.0.stats
dev.bge.0.stats.FramesDroppedDueToFilters: 0
dev.bge.0.stats.DmaWriteQueueFull: 84072
dev.bge.0.stats.DmaWriteHighPriQueueFull: 0
dev.bge.0.stats.NoMoreRxBDs: 0
dev.bge.0.stats.InputDiscards: 0
dev.bge.0.stats.InputErrors: 30
dev.bge.0.stats.RecvThresholdHit: 745400662
dev.bge.0.stats.DmaReadQueueFull: 2020586592
dev.bge.0.stats.DmaReadHighPriQueueFull: 0
dev.bge.0.stats.SendDataCompQueueFull: 0
dev.bge.0.stats.RingSetSendProdIndex: 2832885493
dev.bge.0.stats.RingStatusUpdate: 899990835
dev.bge.0.stats.Interrupts: 899990835
dev.bge.0.stats.AvoidedInterrupts: 0
dev.bge.0.stats.SendThresholdHit: 0
dev.bge.0.stats.rx.ifHCInOctets: 491268800
dev.bge.0.stats.rx.Fragments: 234
dev.bge.0.stats.rx.UnicastPkts: 1977202324
dev.bge.0.stats.rx.MulticastPkts: 0
dev.bge.0.stats.rx.FCSErrors: 341
dev.bge.0.stats.rx.AlignmentErrors: 0
dev.bge.0.stats.rx.xonPauseFramesReceived: 0
dev.bge.0.stats.rx.xoffPauseFramesReceived: 0
dev.bge.0.stats.rx.ControlFramesReceived: 0
dev.bge.0.stats.rx.xoffStateEntered: 0
dev.bge.0.stats.rx.FramesTooLong: 0
dev.bge.0.stats.rx.Jabbers: 0
dev.bge.0.stats.rx.UndersizePkts: 0
dev.bge.0.stats.rx.inRangeLengthError: 0
dev.bge.0.stats.rx.outRangeLengthError: 0
dev.bge.0.stats.tx.ifHCOutOctets: 683592718
dev.bge.0.stats.tx.Collisions: 0
dev.bge.0.stats.tx.XonSent: 0
dev.bge.0.stats.tx.XoffSent: 0
dev.bge.0.stats.tx.flowControlDone: 0
dev.bge.0.stats.tx.InternalMacTransmitErrors: 0
dev.bge.0.stats.tx.SingleCollisionFrames: 0
dev.bge.0.stats.tx.MultipleCollisionFrames: 0
dev.bge.0.stats.tx.DeferredTransmissions: 0
dev.bge.0.stats.tx.ExcessiveCollisions: 0
dev.bge.0.stats.tx.LateCollisions: 0
dev.bge.0.stats.tx.UnicastPkts: 3292778353
dev.bge.0.stats.tx.MulticastPkts: 0
dev.bge.0.stats.tx.BroadcastPkts: 147
dev.bge.0.stats.tx.CarrierSenseErrors: 0
dev.bge.0.stats.tx.Discards: 0
dev.bge.0.stats.tx.Errors: 0

>> Should I try to play with constants in the code and if so, what are limits of this chip?
>> Or it will never be capable of utilizing full gigabit speed?
> 
> BCM5704 is old controller but it should have no problems to
> saturate the link.  I suspect it could be related with DMA
> configuration but needs more information.

I'll supply any information. It's also possible to try kernel patches, if any :-)

Eugene Grosbein