Re: Performance issues with vnet jails + epair + bridge

I just did a throughput test with iperf3 client on a FreeBSD 14.1 host with
an intel 10GB nic connecting to an iperf3 server running in a vnet jail on
a truenas host (13.something) also with an intel 10GB nic and I get full
10GB throughput in this setup. In the past, I had to disable LRO on the
truenas host for this to work properly.

Doug.



On Sat, 14 Sept 2024 at 11:25, Sad Clouds <cryintothebluesky@gmail.com>
wrote:

> On Sat, 14 Sep 2024 10:45:03 +0800
> Zhenlei Huang <zlei@FreeBSD.org> wrote:
>
> > The overhead of vnet jail should be neglectable, compared to legacy jail
> > or no-jail. Bare in mind when VIMAGE option is enabled, there is a
> default
> > vnet 0. It is not visible via jls and can not be destroyed. So when you
> see
> > bottlenecks, for example this case, it is mostly caused by other
> components
> > such as if_epair, but not the vnet jail itself.
>
> Perhaps this needs a correction - the vnet itself may be OK, but due to
> a single physical NIC on this appliance, I cannot use vnet jails
> without virtualised devices like if_epair(4) and if_bridge(4). I think
> there may be other scalability bottlenecks.
>
> I have a similar setup on Solaris
>
> Here devel is a Solaris zone with exclusive IP configuration, which I
> think may be similar to FreeBSD vnet. It has a virtual NIC devel/net0
> which operates over the physical NIC also called net0 in the global
> zone:
>
> $ dladm
> LINK                CLASS     MTU    STATE    OVER
> net0                phys      1500   up       --
> net1                phys      1500   up       --
> net2                phys      1500   up       --
> net3                phys      1500   up       --
> pkgsrc/net0         vnic      1500   up       net0
> devel/net0          vnic      1500   up       net0
>
> If I run TCP bulk data benchmark with 64 concurrent threads, 32
> threads with server process in the global zone and 32 threads with
> client process in the devel zone, then the system evenly spreads the
> load across all CPU cores and none of them are sitting idle:
>
> $ mpstat -A core 1
>  COR minf mjf xcal  intr ithr  csw icsw migr smtx  srw  syscl  usr sys  st
> idl sze
>    0    0   0 2262  2561    4 4744 2085  209 7271    0 747842  272 528
>  0   0   8
>    1    0   0 3187  4209    2 9102 3768  514 10605   0 597012  221 579
>  0   0   8
>    2    0   0 2091  3251    7 6768 2884  307 9557    0 658124  244 556
>  0   0   8
>    3    0   0 1745  1786   16 3494 1520  176 8847    0 746373  273 527
>  0   0   8
>    4    0   0 2797  2767    3 5908 2414  371 7849    0 692873  253 547
>  0   0   8
>    5    0   0 2782  2359    5 4857 2012  324 9431    0 684840  251 549
>  0   0   8
>    6    0   0 4324  4133    0 9138 3592  538 12525   0 516342  191 609
>  0   0   8
>    7    0   0 2180  3249    0 6960 2926  321 8825    0 697861  257 543
>  0   0   8
>
> With FreeBSD I tried "options RSS" and increasing "net.isr.maxthreads"
> however this resulted in some really flaky kernel behavior. So I'm
> thinking that if_epair(4) may be OK for some low-bandwidth use cases,
> i.e. testing firewall rules, etc, but not suitable for things like
> file/object storage servers, etc.
>
>