ixl 40G bad performance?

[Eggert, Lars]

Hi,

On 2015-10-20, at 10:24, Ian Smith <smithi at nimnet.asn.au> wrote:
> Actually, you want to set hw.acpi.cpu.cx_lowest=C1 instead.

Done.

On 2015-10-19, at 17:55, Luigi Rizzo <rizzo at iet.unipi.it> wrote:
> On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars <lars at netapp.com> wrote:
>> The only other sysctls in ixl(4) that look relevant are:
>> 
>>     hw.ixl.rx_itr
>>             The RX interrupt rate value, set to 8K by default.
>> 
>>     hw.ixl.tx_itr
>>             The TX interrupt rate value, set to 4K by default.
>> 
> 
> yes those. raise to 20-50k and see what you get in
> terms of ping latency.

While ixl(4) talks about 8K and 4K, the defaults actually seem to be:

hw.ixl.tx_itr: 122
hw.ixl.rx_itr: 62

Doubling those values *increases* flood ping latency to ~200 usec (from ~116 usec).

Halving them to 62/31 decreases flood ping latency to ~50 usec, but still doesn't increase iperf throughput (still 2.8 Gb/s). Going to 31/16 further drops latency to 24 usec, with no change in throughput.

(Looking at the "interrupt Moderation parameters" #defines in sys/dev/ixl/ixl.h it seems that ixl likes to have its irq rates specified with some weird divider scheme.)

With 5/5 (which corresponds to IXL_ITR_100K), I get down to 16 usec. Unfortunately, throughput is then also down to about 2 Gb/s.

One thing I noticed in top is that one queue irq is using quite a bit of CPU when I run iperf:

   11      0   -92    -     0K  1152K CPU2    2   0:19  50.98% intr{irq293: ixl1:q2}
   11      0   -92    -     0K  1152K WAIT    3   0:02   5.18% intr{irq294: ixl1:q3}
    0      0   -92    0     0K  8944K -      25   0:01   1.07% kernel{ixl1 que}
   11      0   -92    -     0K  1152K WAIT    1   0:01   0.00% intr{irq292: ixl1:q1}
   11      0   -92    -     0K  1152K WAIT    0   0:00   0.00% intr{irq291: ixl1:q0}
    0      0   -92    0     0K  8944K -      22   0:00   0.00% kernel{ixl1 adminq}
    0      0   -92    0     0K  8944K -      31   0:00   0.00% kernel{ixl1 que}
    0      0   -92    0     0K  8944K -      31   0:00   0.00% kernel{ixl1 que}
    0      0   -92    0     0K  8944K -      31   0:00   0.00% kernel{ixl1 que}
   11      0   -92    -     0K  1152K WAIT   -1   0:00   0.00% intr{irq290: ixl1:aq}

With 10G ix interfaces and a throughput of ~9Gb/s, the CPU load is much lower:

   11      0   -92    -     0K  1152K WAIT    0   0:05   7.67% intr{irq274: ix0:que }
    0      0   -92    0     0K  8944K -      27   0:00   0.29% kernel{ix0 que}
    0      0   -92    0     0K  8944K -      10   0:00   0.00% kernel{ix0 linkq}
   11      0   -92    -     0K  1152K WAIT    1   0:00   0.00% intr{irq275: ix0:que }
   11      0   -92    -     0K  1152K WAIT    3   0:00   0.00% intr{irq277: ix0:que }
   11      0   -92    -     0K  1152K WAIT    2   0:00   0.00% intr{irq276: ix0:que }
   11      0   -92    -     0K  1152K WAIT   18   0:00   0.00% intr{irq278: ix0:link}
    0      0   -92    0     0K  8944K -       0   0:00   0.00% kernel{ix0 que}
    0      0   -92    0     0K  8944K -       0   0:00   0.00% kernel{ix0 que}
    0      0   -92    0     0K  8944K -       0   0:00   0.00% kernel{ix0 que}

Lars
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 273 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20151020/6ce6604a/attachment.bin>