Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

[Christian M]

I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF
cheksum offloading is turned off, and -txcsum for xn0 for both VM's.

I feel the throughput is more consistent now, not all over the place as
before, even between runs. But the Retr column (tcp retries) in iperf3 has
jumped up considerably from hundreds/s to thousands/s.

Just a reminder, I have tested this with 11.0-RELEASE also, where the issue
appeared first for me. 10.4-RELEASE is as fast as I could expect it to be,
and 0 retries.

12.0-STABLE:

Connecting to host 172.31.16.128, port 5201
[  5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec  2401   2.85 KBytes

[  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec  3120   17.0 KBytes

[  5]   2.00-3.00   sec   121 MBytes  1.02 Gbits/sec  3203   69.8 KBytes

[  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  3126   15.6 KBytes

[  5]   4.00-5.00   sec   110 MBytes   921 Mbits/sec  2890   15.6 KBytes

[  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec  3308   17.0 KBytes

[  5]   6.00-7.00   sec   104 MBytes   869 Mbits/sec  3046   48.2 KBytes

[  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec  2845   2.85 KBytes

[  5]   8.00-9.00   sec   104 MBytes   874 Mbits/sec  2711   86.8 KBytes

[  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec  2696   14.2 KBytes

[  5]  10.00-11.00  sec   103 MBytes   864 Mbits/sec  2660   31.3 KBytes

[  5]  11.00-12.00  sec  98.8 MBytes   828 Mbits/sec  2476   19.9 KBytes

[  5]  12.00-13.00  sec  99.9 MBytes   838 Mbits/sec  2857   11.3 KBytes

[  5]  13.00-14.00  sec   107 MBytes   894 Mbits/sec  2685   24.1 KBytes

[  5]  14.00-15.00  sec   114 MBytes   953 Mbits/sec  2321   25.5 KBytes

[  5]  15.00-16.00  sec  93.1 MBytes   781 Mbits/sec  2427   48.3 KBytes

[  5]  16.00-17.00  sec   107 MBytes   895 Mbits/sec  2219   29.8 KBytes

[  5]  17.00-18.00  sec  92.5 MBytes   776 Mbits/sec  2441   12.8 KBytes

[  5]  18.00-19.00  sec   116 MBytes   976 Mbits/sec  2840   38.2 KBytes

[  5]  19.00-20.00  sec   102 MBytes   853 Mbits/sec  2573   43.9 KBytes

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.05 GBytes   882 Mbits/sec  54845
sender
[  5]   0.00-20.00  sec  2.05 GBytes   881 Mbits/sec
 receiver

10.3-RELEASE (offloading enabled everywhere for these VM's VIF, and for
xn0):

Connecting to host 172.31.16.122, port 5201
[  5] local 172.31.16.121 port 20712 connected to 172.31.16.122 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   832 MBytes  6.98 Gbits/sec    0    580 KBytes

[  5]   1.00-2.00   sec  1.24 GBytes  10.6 Gbits/sec    0   1.04 MBytes

[  5]   2.00-3.00   sec  1.34 GBytes  11.5 Gbits/sec    0   1.57 MBytes

[  5]   3.00-4.00   sec  1.41 GBytes  12.1 Gbits/sec    0   1.76 MBytes

[  5]   4.00-5.00   sec  1.37 GBytes  11.7 Gbits/sec    0   1.77 MBytes

[  5]   5.00-6.00   sec  1.34 GBytes  11.5 Gbits/sec    0   1.77 MBytes

[  5]   6.00-7.00   sec  1.37 GBytes  11.8 Gbits/sec    0   1.77 MBytes

[  5]   7.00-8.00   sec  1.39 GBytes  12.0 Gbits/sec    0   1.77 MBytes

[  5]   8.00-9.00   sec  1.35 GBytes  11.6 Gbits/sec    0   1.77 MBytes

[  5]   9.00-10.00  sec  1.42 GBytes  12.2 Gbits/sec    0   1.77 MBytes

[  5]  10.00-11.00  sec  1.34 GBytes  11.5 Gbits/sec    0   1.77 MBytes

[  5]  11.00-12.00  sec  1.32 GBytes  11.4 Gbits/sec    0   1.77 MBytes

[  5]  12.00-13.00  sec  1.34 GBytes  11.6 Gbits/sec    0   1.77 MBytes

[  5]  13.00-14.00  sec  1.49 GBytes  12.8 Gbits/sec    0   1.77 MBytes

[  5]  14.00-15.00  sec  1.36 GBytes  11.7 Gbits/sec    0   1.77 MBytes

[  5]  15.00-16.00  sec  1.37 GBytes  11.7 Gbits/sec    0   1.77 MBytes

[  5]  16.00-17.00  sec  1.48 GBytes  12.7 Gbits/sec    0   1.77 MBytes

[  5]  17.00-18.00  sec  1.36 GBytes  11.7 Gbits/sec    0   1.77 MBytes

[  5]  18.00-19.00  sec  1.47 GBytes  12.6 Gbits/sec    0   1.77 MBytes

[  5]  19.00-20.00  sec  1.36 GBytes  11.7 Gbits/sec    0   1.77 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  26.9 GBytes  11.6 Gbits/sec    0             sender
[  5]   0.00-20.01  sec  26.9 GBytes  11.6 Gbits/sec
 receiver