Chelsio NETMAP performance

[Slawa Olhovchenkov]

On Mon, Feb 03, 2020 at 01:39:52PM -0800, Navdeep Parhar wrote:

> On 2/3/20 12:17 PM, Slawa Olhovchenkov wrote:
> > I am try to use Chelsio T540-CR in netmap mode and see poor (compared
> > to Intel 82599ES) performance.
> 
> What approximate FreeBSD version is this?

12.1-STABLE

> > 
> > Same application ac receive only about 8.9Mpss, compared to 12.5Mpps
> > at Intel.
> > 
> > pmc profile show mostly time spend in:
> > 
> > 49.76%  [17802]    service_nm_rxq @ /boot/kernel/if_cxgbe.ko
> >  100.0%  [17802]     t4_vi_intr
> >   100.0%  [17802]      ithread_loop @ /boot/kernel/kernel
> >    100.0%  [17802]       fork_exit
> > 
> > 
> > to be exact at line
> > 
> >         while ((d->rsp.u.type_gen & F_RSPD_GEN) == nm_rxq->iq_gen) {
> > 
> > Is this maximum limit for this vendor?
> 
> No, a T540 should be able to sink full 10Gbps (14.88Mpps) on a single rx
> queue.  Try adding this to your loader.conf:
> 
> hw.cxgbe.toecaps_allowed="0"
> 
> Then try simple netmap "pkt-gen -f rx" instead of any custom app and see
> how many pps it's able to sink.

Thanks! `hw.cxgbe.toecaps_allowed="0"` allow recive 14Mpps for may
application too!

Now I am got only 10% less performance compared to Intel, as I see by
higher Chelsio interrupt cpu time (top show about 30% for every
interrupt handler). Is this normal? Is this posible to optimize?

This is interrupt rate I see:

54651 t5nex0:3b0
56475 t5nex0:3b1
55181 t5nex0:3b2
56577 t5nex0:3b3