ixl 40G bad performance?

[Eggert, Lars]

Hi,

On 2015-10-19, at 16:20, Luigi Rizzo <rizzo at iet.unipi.it> wrote:
> 
> i would look at the following:
> - c states and clock speed - make sure you never go below C1,
>  and fix the clock speed to max.
>  Sure these parameters also affect the 10G card, but there
>  may be strange interaction that trigger the power saving
>  modes in different ways

I already have powerd_flags="-a max -b max -n max" in rc.conf, which I hope should be enough.

> - interrupt moderation (may affect ping latency,
>  do not remember how it is set in ixl but probably a sysctl

ixl(4) describes two sysctls that sound like they control AIM, and they default to off:

hw.ixl.dynamic_tx_itr: 0
hw.ixl.dynamic_rx_itr: 0

> - number of queues (32 is a lot i wouldn't use more than 4-8),
>  may affect cpu-socket affinity

With hw.ixl.max_queues=4 in loader.conf, performance is still unchanged.

> - tso and flow director - i have seen bad effects of
>  accelerations so i would run the iperf test with
>  of these features disabled on both sides, and then enable
>  them one at a time

No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro".

How do I turn off flow director?

> - queue sizes - the driver seems to use 1024 slots which is
>  about 1.5 MB queued, which in turn means you have 300us
>  (and possibly half of that) to drain the queue at 40Gbit/s.
>  150-300us may seem an eternity, but if a couple of cores fall
>  into c7 your budget is gone and the loss will trigger a
>  retransmission and window halving etc.

Also no change with "hw.ixl.ringsz=256" in loader.conf.

This is really weird.

Lars
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 273 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20151019/0013ddbf/attachment.bin>