igb dual-port adapter 1200Mbps limit - what to tune?

[Eugene Perevyazko]

On Wed, Nov 10, 2010 at 01:04:28PM +0200, Eugene Perevyazko wrote:
> 
> I have a router running RELENG_7 with two dual-port igbs - igb0 and igb1 are on
> 82575 on intel s5520ur mb and igb2 and igb3 are on 82576 on ET dual-port card.
> 82576 is in 8x slot.
> Main traffic flows from igb0+igb1 to igb2+igb3, less traffic goes back.
> There's no traffic flow in directions igb0 - igb1 and igb2 - igb3.
> 
> There are vlans on all interfaces.
> 
> igb0 and igb1 are outbound links.
> igb2 and igb3 are connected to switch.
> CPU is E5620 at 2.4GHz, 8 cores, irqs bound to different cores skipping HT ones.
> Tried 2 queues and 1 queue per iface, neither hitting cpu limit. 
> 
> The problem is that traffic through igb2+igb3 is limited at around 1200Mbps Tx
> while I was hoping for 1600-1800Mbps Tx.
> 

I'd like to say that now this host is forwarding 1710 Mb/s @ 189 kpps in one direction + 411 Mb/s @ 140 kpps in reverse direction (30 minutes average).
The following changes were made:
 - no lagg in use
 - only one vlan left instead of 5 vlans
 - added motherboard module 'HP NC360T PCIe DP Gigabit Server Adapter (n1e5132)' with 2 em interfaces (only one used)
 - 3 interfaces are connected to hosts with patchcords, not through switch
 - hyperthreading turned off in bios
 - igb driver patched to reduce irq rate and make it tunable (patch follows)
 - hw.igb.num_queues=4, dev.igb.*.enable_aim=0, dev.igb.*.default_intrrate=4000 the last sysctl is added in patch)
 - igb queues manually repinned to different cores 
 - ipfw rules minimized

igb driver used is 1.9.6/RELENG_7, igb.c version 1.3.2.12

Can't say for sure whether there is a key change or is it all changes summed up,
sorry.

-- 
Eugene Perevyazko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: igb.intrrate.patch
Type: text/x-diff
Size: 1453 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20101201/cff62f5c/igb.intrrate.bin