FreeBSD 8.0 ixgbe Poor Performance

Thu Apr 22 18:20:54 UTC 2010

Welcome, glad to have helped.

Jack

On Thu, Apr 22, 2010 at 11:06 AM, Stephen Sanders
<ssanders at softhammer.net>wrote:

>  Adding "-P 2 " to the iperf client got the rate up to what it should be.
> Also, running multiple tcpreplay's pushed the rate up as well.
>
> Thanks again for the pointers.
>
>
> On 4/22/2010 12:39 PM, Jack Vogel wrote:
>
> Couple more things that come to mind:
>
> make sure you increase mbuf pool, nmbclusters up to at least 262144, and
> the driver uses 4K clusters if
> you go to jumbo frames (nmbjumbop). some workloads will benefit from
> increeasing the various sendspace
> and recvspace parameters, maxsockets and maxfiles are other candidates.
>
> Another item: look in /var/log/messages to see if you are getting any
> Interrupt storm messages, if you are
> that can throttle the irq and reduce performance, there is an
> intr_storm_threshold that you can increase to
> keep that from happening.
>
> Finally, it is sometimes not possible to fully utilize the hardware from a
> single process, you can get limited
> by the socket layer, stack, scheduler, whatever, so you might want to use
> multiple test processes. I believe
> iperf has a builtin way to do this. Run more threads and look at your
> cumulative.
>
> Good luck,
>
> Jack
>
>
> On Thu, Apr 22, 2010 at 8:41 AM, Stephen Sanders <ssanders at softhammer.net>wrote:
>
>>  I believe that "pciconf -lvc" showed that the cards were in the correct
>> slot.  I'm not sure as to what all of the output means but I'm guessing that
>> " cap 10[a0] = PCI-Express 2 endpoint max data 128(256) link x8(x8)" means
>> that the card is an 8 lane card and is using all 8 lanes.
>>
>> Setting  kern.ipc.maxsockbuf to16777216 got a better result with ipref TCP
>> testing.  The rate when from ~2.5Gpbs to ~5.5Gbps.
>>
>> Running iperf in UDP test mode is still yielding ~2.5Gbps.  Running
>> tcpreplay tests is also yielding ~2.5Gbps as well.
>>
>> Command lines for iperf testing are:
>>
>> ipref -t 10 -w 2.5m -l 2.5m -c 169.1.0.2
>> iperf -s -w 2.5m -B 169.1.0.2
>>
>> iperf -t 10 -w 2.5m  -c 169.1.0.2 -u
>> iperf -s -w 2.5m -B 169.1.0.2 -u
>>
>> For the tcpdump test, I'm sending output to /dev/null and using the cache
>> flag on tcpreplay in order to avoid limiting my network interface throughput
>> to the disk speed.
>> Commands lines for this test are:
>>
>> tcpdump -i ix1 -w /dev/null
>> tcpreplay -i ix1 -t -l 0 -K ./rate.pcap
>>
>> Please forgive my lack of kernel building prowess but I'm guessing that
>> the latest driver needs to be built in a FreeBSD STABLE tree.   I ran into
>> an undefined symbol "drbr_needs_enqueue" in the ixgbe code I downloaded.
>>
>> Thanks for all the help.
>>
>> On 4/21/2010 4:53 PM, Jack Vogel wrote:
>>
>> Use my new driver and it will tell you when it comes up with the slot
>> speed is,
>> and if its substandard it will SQUAWK loudly at you :)
>>
>> I think the S5000PAL only has Gen1 PCIE slots which is going to limit you
>> somewhat. Would recommend a current generation (x58 or 5520 chipset)
>> system if you want the full benefit of 10G.
>>
>> BTW, you dont way what adapter, 82598 or 82599, you are using?
>>
>> Jack
>>
>>
>> On Wed, Apr 21, 2010 at 12:52 PM, Stephen Sanders <
>> ssanders at softhammer.net> wrote:
>>
>>> I'd be most pleased to get near 9k.
>>>
>>> I'm running FreeBSD 8.0 amd64 on both of the the test hosts.  I've reset
>>> the configurations to system default as I was getting no where with
>>> sysctl and loader.conf settings.
>>>
>>> The motherboards have been configured to do MSI interrupts.  The
>>> S5000PAL has a MSI to old style interrupt BIOS setting that confuses the
>>> driver interrupt setup.
>>>
>>> The 10Gbps cards should be plugged into the 8x PCI-E slots on both
>>> hosts.  I'm double checking that claim right now and will get back later.
>>>
>>> Thanks
>>>
>>>
>>> On 4/21/2010 2:13 PM, Jack Vogel wrote:
>>> > When you get into the 10G world  your performance will only be as good
>>> > as your weakest link, what I mean is if you connect to something that
>>> has
>>> > less than stellar bus and/or memory performance it is going to throttle
>>> > everything.
>>> >
>>> > Running back to back with two good systems you should be able to get
>>> > near line rate (9K range).  Things that can effect that:  64 bit
>>> kernel,
>>> > TSO, LRO, how many queues come to mind.  The default driver config
>>> > should get you there, so tell me more about your hardware/os config??
>>> >
>>> > Jack
>>> >
>>> >
>>> >
>>> > On Wed, Apr 21, 2010 at 8:04 AM, Brandon Gooch
>>> > <jamesbrandongooch at gmail.com>wrote:
>>> >
>>> >
>>> >> On Wed, Apr 21, 2010 at 9:32 AM, Stephen Sanders
>>> >> <ssanders at softhammer.net> wrote:
>>> >>
>>> >>> I am running speed tests on a pair of systems equipped with Intel
>>> 10Gbps
>>> >>> cards and am getting poor performance.
>>> >>>
>>> >>> iperf and tcpdump testing indicates that the card is running at
>>> roughly
>>> >>> 2.5Gbps max transmit/receive.
>>> >>>
>>> >>> My attempts at turning fiddling with netisr, polling, and varying the
>>> >>> buffer sizes has been fruitless.  I'm sure there is something that
>>> I'm
>>> >>> missing so I'm hoping for suggestions.
>>> >>>
>>> >>> There are two systems that are connected head to head via  cross over
>>> >>> cable.  The two systems have the same hardware configuration.  The
>>> >>> hardware is as follows:
>>> >>>
>>> >>> 2 Intel E5430 (Quad core) @ 2.66 Ghz
>>> >>> Intel S5000PAL Motherboard
>>> >>> 16GB Memory
>>> >>>
>>> >>> My iperf command line for the client is:
>>> >>>
>>> >>> iperf -t 10 -c 169.0.0.1 -w 2.5M -l 2.5M
>>> >>>
>>> >>> My TCP dump test command lines are:
>>> >>>
>>> >>> tcpdump -i ix0 -w/dev/null
>>> >>> tcpreplay -i ix0 -t -l 0 -K ./test.pcap
>>> >>>
>>> >> If you're running 8.0-RELEASE, you might try updating to 8-STABLE.
>>> >> Jack Vogel recently committed updated Intel NIC driver code:
>>> >>
>>> >> http://svn.freebsd.org/viewvc/base/stable/8/sys/dev/ixgbe/
>>> >>
>>> >> -Brandon
>>> >> _______________________________________________
>>> >> freebsd-performance at freebsd.org mailing list
>>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>>> >> To unsubscribe, send any mail to "
>>> >> freebsd-performance-unsubscribe at freebsd.org"
>>> >>
>>> >>
>>> > _______________________________________________
>>> > freebsd-performance at freebsd.org mailing list
>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>>> > To unsubscribe, send any mail to "
>>> freebsd-performance-unsubscribe at freebsd.org"
>>> >
>>> >
>>>
>>>
>>
>>
>
>