Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

[Christian M]

I've made two tests while running tcpdump on the xcp-ng host. I'm not at
all qualified to interpret the .pcap files from tcpdump, but I've put them
on Google Drive and linked them below the two tests. Perhaps someone more
qualified could have a look for anything useful in there. Please note the
extremely uneven throughput for test 2 below. It's almost like the
throughput increased when running tcpdump simultaneously.

Host: XCP-ng 7.6.0
Network: Private Network on host, not connected to any PIF.
VM1: 12.0-RELEASE (1 VIF, 172.31.16.125)
VM2: 12.0-RELEASE (1 VIF, 172.31.15.126)

On the host I listen with tcpdump on the VIF for VM1 in both tests.

VM1 as client:

On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap
xcp-ng-vm1-client.pcap (80M):
https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU
On VM1: iperf3 -c 172.31.16.126
On VM2: iperf3 -s

VM1 iperf3 output:
Connecting to host 172.31.16.126, port 5201
[  5] local 172.31.16.125 port 18182 connected to 172.31.16.126 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  8.00 MBytes  67.1 Mbits/sec  156   15.6 KBytes
[  5]   1.00-2.00   sec  14.1 MBytes   118 Mbits/sec  318   1.43 KBytes
[  5]   2.00-3.00   sec  7.51 MBytes  63.1 Mbits/sec  218   1.43 KBytes
[  5]   3.00-4.00   sec  8.29 MBytes  69.3 Mbits/sec  193   1.43 KBytes
[  5]   4.00-5.01   sec  10.7 MBytes  89.4 Mbits/sec  252   1.43 KBytes
[  5]   5.01-6.00   sec  13.6 MBytes   115 Mbits/sec  313   31.3 KBytes
[  5]   6.00-7.00   sec  8.41 MBytes  70.2 Mbits/sec  309   1.43 KBytes
[  5]   7.00-8.01   sec  12.6 MBytes   106 Mbits/sec  223   1.43 KBytes
[  5]   8.01-9.00   sec  12.5 MBytes   106 Mbits/sec  227   1.43 KBytes
[  5]   9.00-10.00  sec  13.5 MBytes   113 Mbits/sec  263   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   109 MBytes  91.6 Mbits/sec  2472
sender
[  5]   0.00-10.01  sec   109 MBytes  91.3 Mbits/sec
 receiver


VM1 as server:

On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-server.pcap
xcp-ng-vm1-server.pcap: (337M):
https://drive.google.com/open?id=1N3-GccXaBc6hlzFrgshi2vhcCt8vekxF
On VM1: iperf3 -s
On VM2: iperf3 -c 172.31.16.125

VM1 iperf3 output:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.31.16.126, port 31523
[  5] local 172.31.16.125 port 5201 connected to 172.31.16.126 port 34605
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  10.9 MBytes  91.6 Mbits/sec
[  5]   1.00-2.00   sec  14.1 MBytes   118 Mbits/sec
[  5]   2.00-3.01   sec  4.92 MBytes  41.0 Mbits/sec
[  5]   3.01-4.00   sec  10.4 MBytes  87.9 Mbits/sec
[  5]   4.00-5.00   sec  8.52 MBytes  71.2 Mbits/sec
[  5]   5.00-6.00   sec  22.0 MBytes   185 Mbits/sec
[  5]   6.00-7.00   sec  9.25 MBytes  77.8 Mbits/sec
[  5]   7.00-8.00   sec  70.8 MBytes   594 Mbits/sec
[  5]   8.00-9.00   sec  89.5 MBytes   752 Mbits/sec
[  5]   9.00-10.00  sec   105 MBytes   884 Mbits/sec
[  5]  10.00-10.00  sec   180 KBytes  1.19 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   346 MBytes   290 Mbits/sec
 receiver



Den tis 25 juni 2019 kl 10:21 skrev Roger Pau Monné <roger.pau at citrix.com>:

> On Mon, Jun 24, 2019 at 10:24:09PM +0200, Christian M wrote:
> > Thanks for your input Bjarne.
> >
> > Previously I tested with iperf2, but I have made som tests with iperf3
> now
> > also and noticed there are a lot of "Retr" (TCP retries) in some cases.
> > Went back here to your post and saw that your results also showed a lot
> of
> > retries in some cases. My new tests showed similar results to yours
> > (although not nearly as hi throughput as you have). When Linux is client
> > the retries are 0, with FreeBSD as client the number is > 0.
>
> Can you figure out what caused those retries? Wrong MTU, bad
> checksums?
>
> > I'm not sure
> > if our numbers are considered high though, and something that actually
> is a
> > problem? Although, on a internal network with no external factors like
> > interference, I feel this should always be 0 no matter what?
>
> If properly configured yes, there should be no retries when doing
> intra-VM connections.
>
> Thanks, Roger.
>