Re: bhyve: slow network throughput between guest VM and host (and vice versa)?

Hi,
  The trick to achieve larger TCP throughput between the VM and the host
(or other co-located VM) is to have TCP Segmentation Offload (TSO) and TCP
Checksum Offload enabled along the whole packet data path (virtio-net,
bhyve, if_tap, if_bridge, ...).
When these offloads are in place, your TCP endpoint will be able to pass
around 64K packets (irrespective of virtio interface MTU), never performing
TCP segmentation or checksum computation in software. In practice, you
would have the same effect that you have if you set MTU=64K along the
packet datapath.

This is how Linux is able to achieve ~40 Gbps TCP throughput between
virtio-net VM and host if_tap (or if_bridge) interface.
FreeBSD is still missing such optimizations for bhyve, if_tap and
if_bridge. Partly, this is due to virtio-net offload-related specification
being designed to match Linux kernel packet metadata (struct sk_buff).
FreeBSD metadata is different, and conversion is not cheap, requiring some
packet header inspection.

Cheers,
  Vincenzo

Il giorno gio 30 giu 2022 alle ore 12:46 Nils Beyer <nbe@vkf-renzel.de> ha
scritto:

> Hi,
>
> I've setup a FreeBSD VM and am using a VirtIO network interface within.
> Then I've setup
> an ip address 192.168.0.2/30 to that VirtIO NIC of the guest using:
>
>         ifconfig vtnet0 192.168.0.2/30 up
>
> and on the TAP interface of the host an ip address 192.168.0.1/30 using
>
>         ifconfig tap0 192.168.0.1/30 up
>
> Trying an iperf3-transfer (iperf3-server on the host, iperf3-client on the
> guest) with a TCP window size of
> 128k I only get around 2.45Gbit/s:
>
> > # env LD_LIBRARY_PATH=. ./iperf3 -c 192.168.0.1 -w 128k
> > Connecting to host 192.168.0.1, port 5201
> > [  5] local 192.168.0.2 port 25651 connected to 192.168.0.1 port 5201
> > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > [  5]   0.00-1.00   sec   314 MBytes  2.63 Gbits/sec    0   1.43 MBytes
>
> > [  5]   1.00-2.00   sec   301 MBytes  2.52 Gbits/sec    0   1.43 MBytes
>
> > [  5]   2.00-3.00   sec   264 MBytes  2.21 Gbits/sec    0   1.43 MBytes
>
> > [  5]   3.00-4.00   sec   284 MBytes  2.38 Gbits/sec    0   1.43 MBytes
>
> > [  5]   4.00-5.00   sec   296 MBytes  2.48 Gbits/sec    0   1.43 MBytes
>
> > [  5]   5.00-6.00   sec   279 MBytes  2.34 Gbits/sec    0   1.43 MBytes
>
> > [  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec    0   1.43 MBytes
>
> > [  5]   7.00-8.00   sec   310 MBytes  2.60 Gbits/sec    0   1.43 MBytes
>
> > [  5]   8.00-9.00   sec   302 MBytes  2.53 Gbits/sec    0   1.43 MBytes
>
> > [  5]   9.00-10.00  sec   333 MBytes  2.79 Gbits/sec    0   1.43 MBytes
>
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval           Transfer     Bitrate         Retr
> > [  5]   0.00-10.00  sec  2.89 GBytes  2.49 Gbits/sec    0
>  sender
> > [  5]   0.00-10.00  sec  2.89 GBytes  2.49 Gbits/sec
> receiver
>
>
> Switching the roles (iperf3-server on the guest, iperf3-client on the
> host) with a TCP windows size of 128k,
> I get 4.04Gbit/s:
>
> > #iperf3 -c 192.168.0.2 -w 128k
> > Connecting to host 192.168.0.2, port 5201
> > [  5] local 192.168.0.1 port 56892 connected to 192.168.0.2 port 5201
> > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > [  5]   0.00-1.00   sec   411 MBytes  3.44 Gbits/sec   40    973 KBytes
>
> > [  5]   1.00-2.00   sec   483 MBytes  4.05 Gbits/sec    5   1.03 MBytes
>
> > [  5]   2.00-3.00   sec   507 MBytes  4.26 Gbits/sec    1   1.21 MBytes
>
> > [  5]   3.00-4.00   sec   514 MBytes  4.31 Gbits/sec   15    561 KBytes
>
> > [  5]   4.00-5.00   sec   498 MBytes  4.18 Gbits/sec   10    966 KBytes
>
> > [  5]   5.00-6.00   sec   491 MBytes  4.12 Gbits/sec   19    841 KBytes
>
> > [  5]   6.00-7.00   sec   513 MBytes  4.31 Gbits/sec    0   1.43 MBytes
>
> > [  5]   7.00-8.00   sec   504 MBytes  4.23 Gbits/sec    0   1.43 MBytes
>
> > [  5]   8.00-9.00   sec   459 MBytes  3.85 Gbits/sec    0   1.43 MBytes
>
> > [  5]   9.00-10.00  sec   435 MBytes  3.65 Gbits/sec    0   1.43 MBytes
>
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval           Transfer     Bitrate         Retr
> > [  5]   0.00-10.00  sec  4.70 GBytes  4.04 Gbits/sec   90
>  sender
> > [  5]   0.00-10.00  sec  4.70 GBytes  4.04 Gbits/sec
> receiver
>
>
> Increasing MTU on the Virtio interface and on the TAP interface to 9000
> helps a little bit:
> getting 8.38Gbit/s guest->host and 10.3Gbit/s host->guest.
>
> Increasing TCP windows size to 1024k only produces more retries and does
> nothing on the
> throughput.
>
> Is that expected that I'm not able to get more throughput within the bhyve
> network
> stack (guest <-> host)? I was expecting way more then 10Gbit/s...
>
>
>
> TIA and KR,
> Nils
>
>