Performance issues with Intel Fortville (XL710/ixl(4))

[Pokala, Ravi]

Hi folks,

At Panasas, we are working with the Intel XL710 40G NIC (aka Fortville),
and we're seeing some performance issues w/ 11-CURRENT (r282653).

    Motherboard: Intel S2600KP (aka Kennedy Pass)
    CPU: E5-2660 v3 @ 2.6GHz (aka Haswell Xeon)
        (1 socket x 10 physical cores x 2 SMT threads) = 20 logical cores
    NIC: Intel XL710, 2x40Gbps QSFP, configured in 4x10Gbps mode
    RAM: 4x 16GB DDR4 DIMMs

What we've seen so far:

  - TX performance is pretty consistently lower than RX performance. All
numbers below are for unidrectional tests using `iperf':
        10Gbps links    threads/link    TX Gbps     RX Gbps     TX/RX
        1               1               9.02        9.85        91.57%
        1               8               8.49        9.91        85.67%
        1               16              7.00        9.91        70.63%
        1               32              6.68        9.92        67.40%

  - With multiple active links, both TX and RX performance suffer greatly;
the aggregate bandwidth tops out at about a third of the theoretical
40Gbps implied by 4x 10Gbps.
        10Gbps links    threads/link    TX Gbps     RX Gbps     % of 40Gbps
        4               1               13.39       13.38       33.4%

  - Multi-link bidirectional throughput is absolutely terrible; the
aggregate is less than a tenth of the theoretical 40Gbps.
        10Gbps links    threads/link    TX Gbps     RX Gbps     % of 40Gbps
        4               1               3.83        2.96        9.6% / 7.4%

  - Occasional interrupt storm messages are seen from the IRQs associated
with the NICs. Since that can impact performance, those runs were not
included in the data listed above.

Our questions:

  - How stable is ixl(4) in -CURRENT? By that, we mean both how quickly is
the driver changing, and does the driver cause any system instability?

  - What type of performance have others been getting w/ Fortville? In
40Gbps mode? In 4x10Gbps mode?

  - Does anyone have any tuning parameters they can recommend for this
card?

  - We did our testing w/ 11-CURRENT, but we will initially ship Fortville
running on 10.1-RELEASE or 10.2-RELEASE. The presence of RSS - even though
it is disabled by default - makes the driver back-port non-trivial. Is
there an estimate on when the 11-CURRENT version of the driver (1.4.1)
will get MFCed to 10-STABLE?

My colleagues Lakshmi and Fred (CCed) are working on this; please make
sure to include them if you have any comments.

Thanks,

Ravi