Current state of Intel XL710 40G NIC ixl performance

[Rodney W. Grimes]

> Am 28.03.2018 um 18:38 schrieb Rodney W. Grimes:
> >> Am 28.03.2018 um 06:11 schrieb christian russell:
> >>> I am having trouble getting an Intel XL710-DA2 NIC to get even close to
> >>> line rate.  It is a 4x10 Gbps card.  The box is running FreeBSD 11 (FreeNAS
> >>> in particular).
> >>>
> >>> We have tried both 1.7 and 1.9 driver revisions with similar results.  The
> >>> NVM version is 5.05.  The card is in a confirmed 8x slot on a SuperMicro
> >>> X10DRL-i with two Xeon E5-2600 processors and 256 GB DDR4 RAM.  After
> >>> upping the interrupt threshold to 9000 dmesg doesn't log anything unusual.
> >>>
> >>> We have added the tunes that are standard for 10 Gbps configurations.
> >>>
> >>> On a single-client basis the fastest rates we see are around 5 Gbps.
> >>> Hitting this server from multiple boxes we see peaks of 20 Gbps at the very
> >>> highest.  More frequently things top off around 13 Gbps.  These numbers are
> >>> coming from iperf tests.  We are seeing similar numbers with direct
> >>> point-to-point as well as switched topologies.
> >>>
> >>> These threads from 2015 describe similar issues but fizzled out:
> >>> https://lists.freebsd.org/pipermail/freebsd-net/2015-May/042273.html
> >>> https://lists.freebsd.org/pipermail/freebsd-net/2015-October/043584.html
> >>>
> >>> Is there very particular tuning required to get these cards working at
> >>> proper speed?  Any insights?
> >>>
> >>> >From Googling around it appears frustration with this card and FreeBSD is
> >>> pretty common.
> >>>
> >>> Thanks in advance.
> >>>
> >>> Christian
> >> I can't deliver any special insights but we had many problems with X710
> >> (without L) and Linux.
> >> Did some testing a while ago with OPNsense (based on 11.1) and got line
> >> rate with iperf and single client.
> >> ixl0 in and ixl1 out. So this should be fine. If you like I can send you
> >> the sysctl values to compare.
> > I would be interested in your sysctl values.
> >
> 
> Hm, perhaps I misinterpreted your post. I recreated the lab and from the 
> history I saw
> that it was single client but with 10 streams in parallel. If I reduce 
> to 1 stream I also
> don't come over the 5,6G.
> 
> If you're still interested I can send you the output.

We specifically are interested in the sysctl settings you used
to obtain a "line rate" results using ixl0 in and ixl1 out,
it does not matter if that was 1 stream or 10 stream, we would
just like to see what values you found to work well for you
to compare to what values others may be using/suggesting.

-- 
Rod Grimes                                                 rgrimes at freebsd.org