bge interfaces: poor transmit performance?

[David Siebörger]

I'm experiencing a problem with BCM5721 bge interfaces, which seem to be 
able to receive at almost 1Gbps but can only transmit at =< 560Mbps.

I first noticed the problem on a firewall which routes between a number 
of subnets, all connected as VLAN interfaces on bge0.  In its case, 
transmitted traffic and received traffic should be roughly equal (minus 
those packets which the firewall drops) but I've seen that throughput 
hits a limit at 560Mbps (measured from the interface stats using slurm).

To investigate further, I've connected two of the same machines with a 
flylead and run some iperf tests:

Test: iperf -c172.30.3.x -w256k -t30 -P8

	Client		Server		Result
--------------------------------------------------------
1.	6.2-STABLE	7.0-CURRENT	499 Mbits/sec
2.	7.0-CURRENT	6.2-STABLE	526 Mbits/sec
3.	6.2-STABLE	Linux		500 Mbits/sec
4.	Linux		6.2-STABLE	941 Mbits/sec
5.	Linux		Linux		941 Mbits/sec

Test: iperf -c172.30.3.x -w256k -t30 -P4 -d

	Client		Server		Result
--------------------------------------------------------
6.	6.2-STABLE	7.0-CURRENT	381 & 388 Mbits/sec
7.	7.0-CURRENT	6.2-STABLE	369 & 405 Mbits/sec
8.	6.2-STABLE	Linux		423 & 537 Mbits/sec
9.	Linux		6.2-STABLE	421 & 554 Mbits/sec
10.	Linux		Linux		833 & 830 Mbits/sec

(Hardware: Dell PE860, onboard BCM5721 NICs, 2.4 GHz Xeon 3060 CPU.
FreeBSD tuning: net.inet.ip.fw.enable=0, kern.ipc.maxsockbuf=8192000, 
net.inet.tcp.sendspace=262144, net.inet.tcp.recvspace=262144, WITNESS 
and INVARIANTS disabled on -CURRENT.
Linux: Knoppix 5.1.1 with kernel 2.6.19, using tg3 driver.)

The most interesting result I see there is the difference between tests 
3 and 4: just changing the direction of traffic flow makes a major 
difference to performance.

Possible causes that (I think) have been eliminated:

 * IRQ sharing.  'vmstat -i | grep bge' on all machines looks similar to 
this:
	irq16: bge0                       374558          5
	irq17: bge1                       324860          4

 * Hardware architecture.  If Linux can make it perform well, there's no 
fundamental PCI bus bandwidth limitation or anything like that.

 * Network errors.  'netstat -i' has always shown Ierrs and Oerrs = 0.

 * SMP.  I've tried building SMP and non-SMP kernels and got almost 
idenitical results.

 * HZ.  I've tried kernels with the default HZ=1000 and with HZ=2500.

Does anyone have any ideas as to what could be causing the problem, or 
any other tests I could try that might shed light on the problem?


-- 
David Siebörger
drs at rucus.ru.ac.za