Performance problem using Intel X520-DA2

[Kirk Davis]

>-----Original Message-----
>On 24.01.2012 09:18, Nikolay Denev wrote:
>> On Jan 23, 2012, at 11:39 PM, Marcin Markowski wrote:
>>
>>> Hello,
>>>
>>> This message has been sent to freebsd-performance@ but got the 
>>> information that should contact also with freebsd-net at .
>>>
>>> We use FreeBSD as sniffer (libpcap programs) and we experience 
>>> performance problems when incoming traffic is greater than 7.5Gbps/s.
>>> If we check 'top' we see that first irq from network card is using 
>>> 100% CPU. I've tested this on FreeBSD 8.2-RELEASE and 9.0-RELEASE (on 
>>> 9.0 we can see also kernel thread named {ix0 que} using 100% CPU), 
>>> and both systems behave the same. In logs we see also:
>>> interrupt storm detected on "irq268:"; throttling interrupt source
>>>
>>> Our server platform is Intel SR2600URBRP, 2x Xeon X5650, 6GB RAM and 
>>> NIC Intel X520-DA2.
>>>
>>> I'm not sure if problem is with NIC or motherboard in SR2600URBRP, 
>>> because everything is fine when we use other server configuration:
>>> Intel SR1630GP, 1x Xeon X3450, 8GB RAM, NIC X520-DA2
>>>
>>> My /boot/loader.conf:
>>> kern.ipc.nmbclusters=262144
>>> hw.ixgbe.rxd=2048
>>> hw.ixgbe.txd=2048
>>> hw.ixgbe.num_queues=16
>>>
>>> /etc/sysctl.conf
>>> hw.intr_storm_threshold=10000
>>>

I just finished a bunch of performance tests on this card.  In my case I was trying to get as close to a full 10Gb/s as possible on a Dell R710 but I haven't yet tried any sniffing.

Have you tried turning on LRO on the interface ( ifconfig lro ).  In my case this made a big difference and I can now get 9.41Gb/s without high CPU or interrupt storms.  I am also using Jack's latest driver downloaded from intel (version 2.4.4).  Even the driver in 9.0 was older.

Here is what I have 
/etc/sysctl.conf
# Increase the network buffers
kern.ipc.nmbclusters=262144
kern.ipc.maxsockbuf=4194304
hw.intr_storm_threshold=9000
kern.ipc.nmbjumbop=262144

/boot/loader.conf
ixgbe_load="YES"
hw.ixgbe.txd=4096
hw.ixgbe.rxd=4096

---- Kirk