suboptimal bge(4) BCM5704 performance in RELENG_8

[YongHyeon PYUN]

On Tue, Mar 06, 2012 at 07:54:02PM +0700, Eugene Grosbein wrote:
> Hi!
> 
> Yesterday I've updated old HP ProLiant DL360 G4p to 8.3-PRELELEASE/amd64
> running busy icecast2 server in hope it can saturate 1G bge(4) link.
> 
> This server has PCI-X connected HP NC7782 Gigabit Server Adapter (BCM5704).

Would you show me the output of dmesg(bge(4) and brgphy(4) related
ones)?

> Is it supposed to emit more than 540Mbit/s with average packet size equal to 1300?
> 

Yes.

> netstat shows increasing "Drop" counters:
> 
> # netstat -id
> Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
> bge0   1500 <Link#1>      00:18:fe:86:ab:f4 635577217     0     0 1047897138     0     0 454509
> 
> vmstat shows relatively high but pretty sane interrupt rate:
> 
> # vmstat -i | grep bge
> irq25: bge0                    232949783      10149
> 
> Input traffic is less than 20Mbit/s meantime, output does not go above 540Mbit/s.
> Its mbufs usage is normal (server has 2G RAM):
> 
> # netstat -m
> 13459/15986/29445 mbufs in use (current/cache/total)
> 13322/13112/26434/262144 mbuf clusters in use (current/cache/total/max)
> 13322/13046 mbuf+clusters out of packet secondary zone in use (current/cache)
> 0/104/104/131072 4k (page size) jumbo clusters in use (current/cache/total/max)
> 0/0/0/65536 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/32768 16k jumbo clusters in use (current/cache/total/max)
> 30008K/30636K/60645K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/0/0 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
> 
> I've glanced at bge(4) code and see it uses lots of hardcoded constants
> for rx/tx descriptor rings, for interrupt moderation and for interface FIFO queue.

Firmware has a fixed number of descriptors so increasing them
wouldn't give better numbers.

> And no loader tunnables/sysctls like em/igb have.
> 

Show me the output of "sysctl dev.bge.0.stats".

> Should I try to play with constants in the code and if so, what are limits of this chip?
> Or it will never be capable of utilizing full gigabit speed?

BCM5704 is old controller but it should have no problems to
saturate the link.  I suspect it could be related with DMA
configuration but needs more information.

> 
> Eugene Grosbein