FreeBSD 8.0 ixgbe Poor Performance

Thu Apr 22 15:41:30 UTC 2010

I believe that "pciconf -lvc" showed that the cards were in the correct
slot.  I'm not sure as to what all of the output means but I'm guessing
that " cap 10[a0] = PCI-Express 2 endpoint max data 128(256) link
x8(x8)" means that the card is an 8 lane card and is using all 8 lanes.

Setting  kern.ipc.maxsockbuf to16777216 got a better result with ipref
TCP testing.  The rate when from ~2.5Gpbs to ~5.5Gbps.

Running iperf in UDP test mode is still yielding ~2.5Gbps.  Running
tcpreplay tests is also yielding ~2.5Gbps as well.

Command lines for iperf testing are:

ipref -t 10 -w 2.5m -l 2.5m -c 169.1.0.2
iperf -s -w 2.5m -B 169.1.0.2

iperf -t 10 -w 2.5m  -c 169.1.0.2 -u
iperf -s -w 2.5m -B 169.1.0.2 -u

For the tcpdump test, I'm sending output to /dev/null and using the
cache flag on tcpreplay in order to avoid limiting my network interface
throughput to the disk speed.
Commands lines for this test are:

tcpdump -i ix1 -w /dev/null
tcpreplay -i ix1 -t -l 0 -K ./rate.pcap

Please forgive my lack of kernel building prowess but I'm guessing that
the latest driver needs to be built in a FreeBSD STABLE tree.   I ran
into an undefined symbol "drbr_needs_enqueue" in the ixgbe code I
downloaded.

Thanks for all the help.

On 4/21/2010 4:53 PM, Jack Vogel wrote:
> Use my new driver and it will tell you when it comes up with the slot
> speed is,
> and if its substandard it will SQUAWK loudly at you :)
>
> I think the S5000PAL only has Gen1 PCIE slots which is going to limit you
> somewhat. Would recommend a current generation (x58 or 5520 chipset)
> system if you want the full benefit of 10G.
>
> BTW, you dont way what adapter, 82598 or 82599, you are using?
>
> Jack
>
>
> On Wed, Apr 21, 2010 at 12:52 PM, Stephen Sanders
> <ssanders at softhammer.net <mailto:ssanders at softhammer.net>> wrote:
>
>     I'd be most pleased to get near 9k.
>
>     I'm running FreeBSD 8.0 amd64 on both of the the test hosts.  I've
>     reset
>     the configurations to system default as I was getting no where with
>     sysctl and loader.conf settings.
>
>     The motherboards have been configured to do MSI interrupts.  The
>     S5000PAL has a MSI to old style interrupt BIOS setting that
>     confuses the
>     driver interrupt setup.
>
>     The 10Gbps cards should be plugged into the 8x PCI-E slots on both
>     hosts.  I'm double checking that claim right now and will get back
>     later.
>
>     Thanks
>
>
>     On 4/21/2010 2:13 PM, Jack Vogel wrote:
>     > When you get into the 10G world  your performance will only be
>     as good
>     > as your weakest link, what I mean is if you connect to something
>     that has
>     > less than stellar bus and/or memory performance it is going to
>     throttle
>     > everything.
>     >
>     > Running back to back with two good systems you should be able to get
>     > near line rate (9K range).  Things that can effect that:  64 bit
>     kernel,
>     > TSO, LRO, how many queues come to mind.  The default driver config
>     > should get you there, so tell me more about your hardware/os
>     config??
>     >
>     > Jack
>     >
>     >
>     >
>     > On Wed, Apr 21, 2010 at 8:04 AM, Brandon Gooch
>     > <jamesbrandongooch at gmail.com
>     <mailto:jamesbrandongooch at gmail.com>>wrote:
>     >
>     >
>     >> On Wed, Apr 21, 2010 at 9:32 AM, Stephen Sanders
>     >> <ssanders at softhammer.net <mailto:ssanders at softhammer.net>> wrote:
>     >>
>     >>> I am running speed tests on a pair of systems equipped with
>     Intel 10Gbps
>     >>> cards and am getting poor performance.
>     >>>
>     >>> iperf and tcpdump testing indicates that the card is running
>     at roughly
>     >>> 2.5Gbps max transmit/receive.
>     >>>
>     >>> My attempts at turning fiddling with netisr, polling, and
>     varying the
>     >>> buffer sizes has been fruitless.  I'm sure there is something
>     that I'm
>     >>> missing so I'm hoping for suggestions.
>     >>>
>     >>> There are two systems that are connected head to head via
>      cross over
>     >>> cable.  The two systems have the same hardware configuration.  The
>     >>> hardware is as follows:
>     >>>
>     >>> 2 Intel E5430 (Quad core) @ 2.66 Ghz
>     >>> Intel S5000PAL Motherboard
>     >>> 16GB Memory
>     >>>
>     >>> My iperf command line for the client is:
>     >>>
>     >>> iperf -t 10 -c 169.0.0.1 -w 2.5M -l 2.5M
>     >>>
>     >>> My TCP dump test command lines are:
>     >>>
>     >>> tcpdump -i ix0 -w/dev/null
>     >>> tcpreplay -i ix0 -t -l 0 -K ./test.pcap
>     >>>
>     >> If you're running 8.0-RELEASE, you might try updating to 8-STABLE.
>     >> Jack Vogel recently committed updated Intel NIC driver code:
>     >>
>     >> http://svn.freebsd.org/viewvc/base/stable/8/sys/dev/ixgbe/
>     >>
>     >> -Brandon
>     >> _______________________________________________
>     >> freebsd-performance at freebsd.org
>     <mailto:freebsd-performance at freebsd.org> mailing list
>     >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>     >> To unsubscribe, send any mail to "
>     >> freebsd-performance-unsubscribe at freebsd.org
>     <mailto:freebsd-performance-unsubscribe at freebsd.org>"
>     >>
>     >>
>     > _______________________________________________
>     > freebsd-performance at freebsd.org
>     <mailto:freebsd-performance at freebsd.org> mailing list
>     > http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>     > To unsubscribe, send any mail to
>     "freebsd-performance-unsubscribe at freebsd.org
>     <mailto:freebsd-performance-unsubscribe at freebsd.org>"
>     >
>     >
>
>