Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Christian M
christian.marcos at gmail.com
Thu Jul 4 07:58:23 UTC 2019
Thanks Roger.
I disabled pv nic entirely on my two 12.0-RELEASE test VM's. I got
1000baseT full duplex auto selected, so I expected throughput close to
that, but to my surprise this are the results:
$ ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
ether 56:65:6f:f3:02:fb
inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
$ iperf3 -c 172.31.16.126
Connecting to host 172.31.16.126, port 5201
[ 5] local 172.31.16.125 port 11247 connected to 172.31.16.126 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 6.50 MBytes 54.5 Mbits/sec 0 368 KBytes
[ 5] 1.00-2.00 sec 7.22 MBytes 60.6 Mbits/sec 0 368 KBytes
[ 5] 2.00-3.00 sec 7.26 MBytes 60.9 Mbits/sec 0 368 KBytes
[ 5] 3.00-4.00 sec 7.09 MBytes 59.4 Mbits/sec 0 368 KBytes
[ 5] 4.00-5.00 sec 7.32 MBytes 61.4 Mbits/sec 0 368 KBytes
[ 5] 5.00-6.00 sec 7.22 MBytes 60.6 Mbits/sec 0 385 KBytes
[ 5] 6.00-7.00 sec 7.24 MBytes 60.7 Mbits/sec 0 385 KBytes
[ 5] 7.00-8.00 sec 7.28 MBytes 61.1 Mbits/sec 0 385 KBytes
[ 5] 8.00-9.00 sec 7.55 MBytes 63.3 Mbits/sec 0 385 KBytes
[ 5] 9.00-10.00 sec 7.12 MBytes 59.7 Mbits/sec 0 385 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec 0 sender
[ 5] 0.00-10.06 sec 71.8 MBytes 59.9 Mbits/sec
receiver
iperf Done.
$ iperf3 -c 172.31.16.126 -R
Connecting to host 172.31.16.126, port 5201
Reverse mode, remote host 172.31.16.126 is sending
[ 5] local 172.31.16.125 port 22443 connected to 172.31.16.126 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 5.06 MBytes 42.5 Mbits/sec
[ 5] 1.00-2.00 sec 5.60 MBytes 47.0 Mbits/sec
[ 5] 2.00-3.00 sec 6.22 MBytes 52.2 Mbits/sec
[ 5] 3.00-4.00 sec 6.92 MBytes 58.0 Mbits/sec
[ 5] 4.00-5.00 sec 7.39 MBytes 62.0 Mbits/sec
[ 5] 5.00-6.00 sec 7.17 MBytes 60.1 Mbits/sec
[ 5] 6.00-7.00 sec 7.18 MBytes 60.3 Mbits/sec
[ 5] 7.00-8.00 sec 7.32 MBytes 61.4 Mbits/sec
[ 5] 8.00-9.00 sec 6.94 MBytes 58.2 Mbits/sec
[ 5] 9.00-10.00 sec 7.04 MBytes 59.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 67.3 MBytes 56.5 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 66.8 MBytes 56.1 Mbits/sec
receiver
iperf Done.
Good luck with your talk.
Regards,
Christian
Den tors 4 juli 2019 kl 09:24 skrev Roger Pau Monné <roger.pau at citrix.com>:
> On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote:
> > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné <
> roger.pau at citrix.com>:
> >
> > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote:
> > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and
> .128.
> > > VIF
> > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's.
> > > >
> > > > I feel the throughput is more consistent now, not all over the place
> as
> > > > before, even between runs. But the Retr column (tcp retries) in
> iperf3
> > > has
> > > > jumped up considerably from hundreds/s to thousands/s.
> > > >
> > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the
> > > issue
> > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it
> to
> > > be,
> > > > and 0 retries.
> > > >
> > > > 12.0-STABLE:
> > > >
> > > > Connecting to host 172.31.16.128, port 5201
> > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port
> 5201
> > > > [ ID] Interval Transfer Bitrate Retr Cwnd
> > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85
> KBytes
> > > >
> > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0
> KBytes
> > > >
> > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8
> KBytes
> > > >
> > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6
> KBytes
> > > >
> > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6
> KBytes
> > > >
> > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0
> KBytes
> > > >
> > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2
> KBytes
> > > >
> > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85
> KBytes
> > > >
> > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8
> KBytes
> > > >
> > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2
> KBytes
> > > >
> > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3
> KBytes
> > > >
> > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9
> KBytes
> > > >
> > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3
> KBytes
> > > >
> > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1
> KBytes
> > > >
> > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5
> KBytes
> > > >
> > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3
> KBytes
> > > >
> > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8
> KBytes
> > > >
> > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8
> KBytes
> > > >
> > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2
> KBytes
> > > >
> > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9
> KBytes
> > > >
> > > > - - - - - - - - - - - - - - - - - - - - - - - - -
> > > > [ ID] Interval Transfer Bitrate Retr
> > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845
> > >
> > > Can you paste the output of ifconfig for both the interfaces used in
> > > the test?
> > >
> > > Are you sure all hardware offloading capabilities are turned off on
> > > both interfaces?
> > >
> > > Can you check what's causing those retries?
> > >
> > > Either using tcpdump, whireshark or some other tool to analyze the
> > > network traffic and detect the errors that cause such retries?
> > >
> > > Thanks, Roger.
> > >
> >
> > 172.31.16.127 (12.0-STABLE):
> >
> > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> > inet6 ::1 prefixlen 128
> > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> > inet 127.0.0.1 netmask 0xff000000
> > groups: lo
> > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > options=501<RXCSUM,TSO4,LRO>
>
> I would try to disable rxcsum, tso4 and lro also.
>
> > ether 6e:83:99:ed:ce:f7
> > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255
> > media: Ethernet manual
> > status: active
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> >
> > ethtool -k vif68.0
> > Features for vif68.0:
> > rx-checksumming: on [fixed]
> > tx-checksumming: off
> > tx-checksum-ipv4: off
> > tx-checksum-ip-generic: off [fixed]
> > tx-checksum-ipv6: off
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: off
> > tx-scatter-gather: off
> > tx-scatter-gather-fraglist: off
> > tcp-segmentation-offload: off
> > tx-tcp-segmentation: off
> > tx-tcp-ecn-segmentation: off [fixed]
> > tx-tcp6-segmentation: off
> > udp-fragmentation-offload: off [fixed]
> > generic-segmentation-offload: off
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: off [fixed]
> > tx-vlan-offload: off [fixed]
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: off [fixed]
> > rx-vlan-filter: off [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-ipip-segmentation: off [fixed]
> > tx-sit-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: off [fixed]
> >
> > 172.31.16.128 (12.0-STABLE):
> >
> > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> > inet6 ::1 prefixlen 128
> > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> > inet 127.0.0.1 netmask 0xff000000
> > groups: lo
> > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > options=501<RXCSUM,TSO4,LRO>
> > ether 32:a7:9f:cc:94:8a
> > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255
> > media: Ethernet manual
> > status: active
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> >
> > ethtool -k vif67.0
> > Features for vif67.0:
> > rx-checksumming: on [fixed]
> > tx-checksumming: off
> > tx-checksum-ipv4: off
> > tx-checksum-ip-generic: off [fixed]
> > tx-checksum-ipv6: off
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: off
> > tx-scatter-gather: off
> > tx-scatter-gather-fraglist: off
> > tcp-segmentation-offload: off
> > tx-tcp-segmentation: off
> > tx-tcp-ecn-segmentation: off [fixed]
> > tx-tcp6-segmentation: off
> > udp-fragmentation-offload: off [fixed]
> > generic-segmentation-offload: off
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: off [fixed]
> > tx-vlan-offload: off [fixed]
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: off [fixed]
> > rx-vlan-filter: off [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-ipip-segmentation: off [fixed]
> > tx-sit-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: off [fixed]
> >
> > Should I run tcpdump on xen host, or on FreeBSD while running iperf3?
>
> On the Xen host would be better, since that tcpdump will read the
> packets as FreeBSD puts them on the wire.
>
> > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying
> > around just to see how 12.0-RELEASE behaved there, and It was the same
> > thing, super slow between VM's.
>
> Yes, I don't think this is related to hardware at all.
>
> I will try to find some time to look into this, but it's not going to
> be soon since I'm currently preparing a talk for a conference I'm
> attending next week, so I won't likely be able to get to it until the
> end of July, sorry. Also, I'm no networking expert, so I cannot assure
> my success in fixing this.
>
> As a workaround you can switch to the emulated network card by
> setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will
> give you worse performance than a fully working PV network card, but
> at least should be consistent. There are others that have switched to
> virtio-net, but I have no idea how to do that with XCP.
>
> Roger.
>
More information about the freebsd-xen
mailing list