terrible if_vmx / vmxnet3 rx performance with lro (post iflib)

[Andriy Gapon]

At work we are seeing some rather terrible vmxnet3 rx performance when lro is
enabled.  We have it enabled because with the pre-iflib vmxnet3 it improved the
performance quite significantly.  For example, in a particular test
configuration we could receive ~1 Gbps w/o lro and ~1.8 Gbps with it.
With the latest driver we can see the same ~1 Gbps w/o lro, but enabling lro
kills the performance -- it's ~40 Mbps.

There is one significant difference that I see between the old and the new code.
Previously, vmxnet3 would just communicate to the virtual hardware that it wants
lro enabled, e.g.:
        if (ifp->if_capenable & IFCAP_LRO)
                ds->upt_features |= UPT1_F_LRO;
At the same time, it never used tcp_lro_* routines.

Nowadays, if lro is enabled, then the iflib code would pass eligible packets to
tcp_lro_rx() and so on.

I hacked iflib code to not do that while leaving vmxnet3 code unchanged.
The performance with lro enabled immediately improved to ~1.4 Gbps.
It's still a bit worse than the pre-iflib code, but much better than iflib+lro.

I cannot explain my observations but I confirmed them multiple times.
Perhaps, the way that the "hardware" lro works hits a degenerate case either in
tcp_lro_* code or in how iflib calls that code.

Perhaps, iflib needs a special lro quirk for a driver such vmxnet3?

P.S.
My general experience with post-iflib vmxnet3 is that vmxnet3 has some
peculiarities that result in a certain "impedance mismatch" with iflib.
Although we now have a bit less code and it is a bit more regular, there are a
few significant (for us, at least) problems:
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243126
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240608
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243392
- the problem described above
- a couple of issues that we already fixed or worked around

We are contemplating locally reverting to the pre-iflib vmxnet3 and we are
wondering if the conversion was really worth it in general.

-- 
Andriy Gapon