ixl 40G bad performance?

[Denis Pearson]

On Thu, Dec 10, 2015 at 4:40 PM, Adrian Chadd <adrian.chadd at gmail.com>
wrote:

> On 10 December 2015 at 10:29, Denis Pearson <dennix.pearson at gmail.com>
> wrote:
> > On Thu, Dec 10, 2015 at 2:18 PM, Eggert, Lars <lars at netapp.com> wrote:
> >
> >> On 2015-10-26, at 18:40, Eggert, Lars <lars at netapp.com> wrote:
> >> > On 2015-10-26, at 17:08, Pieper, Jeffrey E <
> jeffrey.e.pieper at intel.com>
> >> wrote:
> >> >> As a caveat, this was using default netperf message sizes.
> >> >
> >> > I get the same ~3 Gb/s with the default netperf sizes and driver
> 1.4.5.
> >>
> >> Now there is version 1.4.8 on the Intel website, but it doesn't change
> >> things for me.
> >>
> >
> > I had the opportunity to see similar numbers and behavior while using
> XL710
> > 1.4.3 as of FreeBSD r291085 while in DPDK poll mode, but driver 1.2.8 as
> of
> > r292035 was providing expected numbers. While removing rxcsum/txcsum did
> > not provide differences, fully removing RSS + disabling rx/cxsum support
> > provided better numbers.
>
> Can someone debug this a bit more? (My kit with ixl NICs in it is
> still not up and available. :( )
>
> Device RSS, even without kernel RSS enabled, shouldn't cause a massive
> performance drop. If it is then something else odd is going on.


> Do you have a diff where you removed things?
>

I can probably find out a snapshot with the code at the time and extract a
diff, yes. I just don't know how it worths wasting the time when the
problem is not reproducible on the current 1.4.8 driver which will
hopefully get into -CURRENT (if it's not already there?). And it's much
more specific, the performance drop happened on dpdk poll mode, not the
usual kernel operation so a simple diff only pointing out the changes for
the driver to actually build and run without rss will still require a
testlab and different ways to generate traffic.

This is why I suggested a transceiver change or replug first.

Anyway RSS performance dropping problem is far from a FreeBSD specific
problem, while researching I could find the exact same complaints on
Windows users starting from windows 8 while having RSS at 4 or RSS at 16 or RSS
completely disabled, some times with acceptable results only when it was
disabled (despiste the fact that MiniportInterruptDPC was using a whole CPU
when RSS was off results were still better). So I guess this is just a side
effect of when it's just good to have NIC features turned off. The reason,
I'm not an engineer to answer, but I would guess it's related to other NIC
features also doing something with the packet or any sort of errors netstat
or driver status may not tell.

I was able to see the problem even with low pps rates and big packet sizes,
as well as avg pkt size of 768bytes so I don't think it's any sort of card
resource starvation. I can manage to have the whole lab up and running by
the weekend if you want to investigate and compare, just ping me off list.


>
> -adrian
>
> > However now with driver 1.4.8 and the same set of hardware setup, except
> > for a different transceiver, I can get 36Gbps/24Mpps with no further
> > tweaks, so if you can replace your transceiver, shall be a different test
> > as a starting point.
>