Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Christian M
christian.marcos at gmail.com
Wed Jun 26 08:12:02 UTC 2019
checksum offloading doe not seem to have any affect when done on the VM
VIF's. Both 12.0-RELEASE VIF's now have:
other-config (MRW): ethtool-rx: off; ethtool-tx: off; ethtool-sg: off;
ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off
ethtool -k vif54.0
Features for vif54.0:
rx-checksumming: on [fixed]
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
Both vif's look the same as above.
Running tcpdump on the host while running iperf3 between the 12.0 VM's
results in a lot of incorrect cksum like this.
tcpdump -i vif54.0 -v -nn| grep -i incorrect
172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08
(incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options
[nop,nop,TS val 4104002274 ecr 3926764642], length 1448
...
Also noticed that iperf3 does not work to run every time, more than 50% of
the tries with iperf3 -c <ip> I get:
Connecting to host 172.31.16.126, port 5201
iperf3: error - unable to initialize stream: Socket is not connected
Or nothing happens and I need to abort, then restart iperf3 -s on the other
VM, and retry.
The results from iperf3 are really inconsistent still, and I cant get a
good feel of any pattern. Currently as I type this, a result looks like
this:
Host: tcpdump running
VIF's: no promisc
xn0 on both VM's: no promisc
[ 5] local 172.31.16.125 port 47738 connected to 172.31.16.126 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 37.8 MBytes 316 Mbits/sec 176 1.43 KBytes
[ 5] 1.00-2.00 sec 133 MBytes 1.11 Gbits/sec 1022 49.8 KBytes
[ 5] 2.00-3.00 sec 179 MBytes 1.50 Gbits/sec 784 191 KBytes
[ 5] 3.00-4.00 sec 132 MBytes 1.11 Gbits/sec 201 345 KBytes
[ 5] 4.00-5.00 sec 95.3 MBytes 799 Mbits/sec 438 180 KBytes
[ 5] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 1256 353 KBytes
[ 5] 6.00-7.00 sec 144 MBytes 1.20 Gbits/sec 1242 275 KBytes
[ 5] 7.00-8.00 sec 94.5 MBytes 792 Mbits/sec 593 167 KBytes
[ 5] 8.00-9.00 sec 125 MBytes 1.05 Gbits/sec 326 169 KBytes
[ 5] 9.00-10.00 sec 57.0 MBytes 478 Mbits/sec 1013 1.43 KBytes
[ 5] 10.00-11.00 sec 31.3 MBytes 262 Mbits/sec 463 1.43 KBytes
[ 5] 11.00-12.00 sec 42.3 MBytes 355 Mbits/sec 544 78.3 KBytes
[ 5] 12.00-13.00 sec 41.3 MBytes 346 Mbits/sec 624 1.43 KBytes
[ 5] 13.00-14.00 sec 52.7 MBytes 442 Mbits/sec 739 1.43 KBytes
[ 5] 14.00-15.00 sec 20.4 MBytes 171 Mbits/sec 384 1.43 KBytes
[ 5] 15.00-16.00 sec 74.5 MBytes 625 Mbits/sec 489 299 KBytes
[ 5] 16.00-17.00 sec 40.1 MBytes 337 Mbits/sec 835 1.43 KBytes
[ 5] 17.00-18.00 sec 43.2 MBytes 363 Mbits/sec 372 1.43 KBytes
[ 5] 18.00-19.00 sec 58.9 MBytes 494 Mbits/sec 723 1.43 KBytes
[ 5] 19.00-20.00 sec 21.2 MBytes 178 Mbits/sec 300 1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec 12524
sender
[ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec
receiver
But please be sure, this can change dramatically without making any changes
what so ever. Offloading is not turned off in FreeBSD (but on the VIF's as
stated before), and tcpdump is running on the host.
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=503<RXCSUM,TXCSUM,TSO4,LRO>
ether 12:ec:6a:5c:af:df
inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet manual
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=503<RXCSUM,TXCSUM,TSO4,LRO>
ether aa:4f:f7:4c:a0:e6
inet 172.31.16.126 netmask 0xffffff00 broadcast 172.31.16.255
media: Ethernet manual
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Absolutely nothing changed this time (restarted iperf3 -s though since I
ran ifconfig do get the info above), tcpdump still running on host (never
even aborted):
Host: tcpdump still running
VIF's: no promisc
xn0 on both VM's: no promisc
[ 5] local 172.31.16.125 port 30543 connected to 172.31.16.126 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.3 MBytes 86.8 Mbits/sec 165 38.4 KBytes
[ 5] 1.00-2.00 sec 39.7 MBytes 333 Mbits/sec 175 167 KBytes
[ 5] 2.00-3.00 sec 40.0 MBytes 335 Mbits/sec 437 54.1 KBytes
[ 5] 3.00-4.00 sec 11.2 MBytes 93.6 Mbits/sec 320 1.43 KBytes
[ 5] 4.00-5.00 sec 11.2 MBytes 94.2 Mbits/sec 305 1.43 KBytes
[ 5] 5.00-6.00 sec 7.63 MBytes 64.1 Mbits/sec 179 2.84 KBytes
[ 5] 6.00-7.00 sec 6.77 MBytes 56.7 Mbits/sec 191 1.43 KBytes
[ 5] 7.00-8.00 sec 4.26 MBytes 35.9 Mbits/sec 145 21.3 KBytes
[ 5] 8.00-9.00 sec 5.38 MBytes 45.1 Mbits/sec 176 1.43 KBytes
[ 5] 9.00-10.00 sec 8.24 MBytes 69.1 Mbits/sec 225 1.43 KBytes
[ 5] 10.00-11.00 sec 16.1 MBytes 135 Mbits/sec 422 1.43 KBytes
[ 5] 11.00-12.00 sec 3.76 MBytes 31.5 Mbits/sec 113 1.43 KBytes
[ 5] 12.00-13.00 sec 6.62 MBytes 55.5 Mbits/sec 161 1.43 KBytes
[ 5] 13.00-14.00 sec 5.28 MBytes 44.3 Mbits/sec 156 18.4 KBytes
[ 5] 14.00-15.00 sec 5.21 MBytes 43.7 Mbits/sec 146 1.43 KBytes
[ 5] 15.00-16.00 sec 6.37 MBytes 53.4 Mbits/sec 199 1.43 KBytes
[ 5] 16.00-17.00 sec 10.7 MBytes 89.5 Mbits/sec 193 1.43 KBytes
[ 5] 17.00-18.00 sec 9.49 MBytes 79.6 Mbits/sec 390 1.43 KBytes
[ 5] 18.00-19.00 sec 5.36 MBytes 44.9 Mbits/sec 125 1.43 KBytes
[ 5] 19.00-20.00 sec 3.65 MBytes 30.6 Mbits/sec 58 1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 217 MBytes 91.1 Mbits/sec 4281
sender
[ 5] 0.00-20.00 sec 217 MBytes 90.8 Mbits/sec
receiver
Host: tcpdump still running
VIF's: promisc
xn0 on both VM's: promisc
Connecting to host 172.31.16.126, port 5201
[ 5] local 172.31.16.125 port 61196 connected to 172.31.16.126 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 25.9 MBytes 217 Mbits/sec 271 90.5 KBytes
[ 5] 1.00-2.00 sec 56.4 MBytes 473 Mbits/sec 94 255 KBytes
[ 5] 2.00-3.00 sec 83.4 MBytes 699 Mbits/sec 872 201 KBytes
[ 5] 3.00-4.00 sec 76.4 MBytes 641 Mbits/sec 684 176 KBytes
[ 5] 4.00-5.00 sec 80.4 MBytes 674 Mbits/sec 742 1.43 KBytes
[ 5] 5.00-6.00 sec 81.4 MBytes 684 Mbits/sec 248 333 KBytes
[ 5] 6.00-7.00 sec 86.1 MBytes 722 Mbits/sec 826 160 KBytes
[ 5] 7.00-8.00 sec 77.5 MBytes 650 Mbits/sec 209 253 KBytes
[ 5] 8.00-9.00 sec 9.64 MBytes 80.9 Mbits/sec 433 12.8 KBytes
[ 5] 9.00-10.00 sec 8.31 MBytes 69.6 Mbits/sec 210 1.43 KBytes
[ 5] 10.00-11.00 sec 8.64 MBytes 72.5 Mbits/sec 199 1.43 KBytes
[ 5] 11.00-12.00 sec 7.02 MBytes 58.9 Mbits/sec 240 1.43 KBytes
[ 5] 12.00-13.00 sec 20.1 MBytes 169 Mbits/sec 579 1.43 KBytes
[ 5] 13.00-14.00 sec 10.8 MBytes 90.4 Mbits/sec 277 22.7 KBytes
[ 5] 14.00-15.00 sec 2.75 MBytes 23.1 Mbits/sec 107 1.43 KBytes
[ 5] 15.00-16.00 sec 8.79 MBytes 73.8 Mbits/sec 203 12.8 KBytes
[ 5] 16.00-17.00 sec 9.23 MBytes 77.3 Mbits/sec 158 1.43 KBytes
[ 5] 17.00-18.00 sec 13.5 MBytes 114 Mbits/sec 381 1.43 KBytes
[ 5] 18.00-19.00 sec 41.6 MBytes 349 Mbits/sec 331 99.5 KBytes
[ 5] 19.00-20.00 sec 40.6 MBytes 340 Mbits/sec 70 1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 749 MBytes 314 Mbits/sec 7134
sender
[ 5] 0.00-20.00 sec 748 MBytes 314 Mbits/sec
receiver
Host: tcpdump NOT running
VIF's: no promisc
xn0: no promisc
Run1:
Connecting to host 172.31.16.126, port 5201
[ 5] local 172.31.16.125 port 20600 connected to 172.31.16.126 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 28.4 MBytes 238 Mbits/sec 451 1.43 KBytes
[ 5] 1.00-2.00 sec 21.5 MBytes 181 Mbits/sec 546 2.84 KBytes
[ 5] 2.00-3.00 sec 26.7 MBytes 224 Mbits/sec 595 1.43 KBytes
[ 5] 3.00-4.00 sec 15.9 MBytes 133 Mbits/sec 347 1.43 KBytes
[ 5] 4.00-5.00 sec 4.45 MBytes 37.4 Mbits/sec 187 2.84 KBytes
[ 5] 5.00-6.00 sec 15.4 MBytes 129 Mbits/sec 239 66.5 KBytes
[ 5] 6.00-7.00 sec 10.6 MBytes 89.1 Mbits/sec 366 1.43 KBytes
[ 5] 7.00-8.00 sec 30.8 MBytes 258 Mbits/sec 510 1.43 KBytes
[ 5] 8.00-9.00 sec 16.6 MBytes 140 Mbits/sec 332 2.84 KBytes
[ 5] 9.00-10.00 sec 12.4 MBytes 104 Mbits/sec 376 1.43 KBytes
[ 5] 10.00-11.00 sec 11.3 MBytes 94.9 Mbits/sec 223 1.43 KBytes
[ 5] 11.00-12.00 sec 9.00 MBytes 75.5 Mbits/sec 206 1.43 KBytes
[ 5] 12.00-13.00 sec 8.95 MBytes 75.1 Mbits/sec 201 1.43 KBytes
[ 5] 13.00-14.00 sec 12.3 MBytes 103 Mbits/sec 179 1.43 KBytes
[ 5] 14.00-15.00 sec 20.5 MBytes 172 Mbits/sec 445 1.43 KBytes
[ 5] 15.00-16.00 sec 3.92 MBytes 32.9 Mbits/sec 163 1.43 KBytes
[ 5] 16.00-17.00 sec 6.82 MBytes 57.2 Mbits/sec 122 2.84 KBytes
[ 5] 17.00-18.00 sec 14.3 MBytes 120 Mbits/sec 238 1.43 KBytes
[ 5] 18.00-19.00 sec 14.2 MBytes 119 Mbits/sec 226 1.43 KBytes
[ 5] 19.00-20.00 sec 12.9 MBytes 108 Mbits/sec 297 1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 297 MBytes 125 Mbits/sec 6249
sender
[ 5] 0.00-20.00 sec 297 MBytes 124 Mbits/sec
receiver
Run2 right after a iperf3 -s restart on second VM:
Connecting to host 172.31.16.126, port 5201
[ 5] local 172.31.16.125 port 40622 connected to 172.31.16.126 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 6.49 MBytes 54.3 Mbits/sec 163 1.43 KBytes
[ 5] 1.00-2.00 sec 9.93 MBytes 83.3 Mbits/sec 263 1.43 KBytes
[ 5] 2.00-3.00 sec 7.71 MBytes 64.7 Mbits/sec 173 1.43 KBytes
[ 5] 3.00-4.00 sec 5.58 MBytes 47.0 Mbits/sec 83 1.43 KBytes
[ 5] 4.00-5.00 sec 14.8 MBytes 124 Mbits/sec 347 1.43 KBytes
[ 5] 5.00-6.00 sec 16.8 MBytes 141 Mbits/sec 349 1.43 KBytes
[ 5] 6.00-7.00 sec 19.2 MBytes 161 Mbits/sec 205 1.43 KBytes
[ 5] 7.00-8.00 sec 6.39 MBytes 53.6 Mbits/sec 79 1.43 KBytes
[ 5] 8.00-9.00 sec 7.19 MBytes 60.3 Mbits/sec 133 1.43 KBytes
[ 5] 9.00-10.00 sec 18.2 MBytes 153 Mbits/sec 265 1.43 KBytes
[ 5] 10.00-11.00 sec 11.0 MBytes 92.6 Mbits/sec 220 1.43 KBytes
[ 5] 11.00-12.00 sec 7.93 MBytes 66.6 Mbits/sec 172 1.43 KBytes
[ 5] 12.00-13.00 sec 10.1 MBytes 84.8 Mbits/sec 198 1.43 KBytes
[ 5] 13.00-14.00 sec 7.92 MBytes 66.5 Mbits/sec 196 1.43 KBytes
[ 5] 14.00-15.00 sec 7.63 MBytes 64.0 Mbits/sec 144 1.43 KBytes
[ 5] 15.00-16.00 sec 8.70 MBytes 73.0 Mbits/sec 267 1.43 KBytes
[ 5] 16.00-17.00 sec 16.0 MBytes 134 Mbits/sec 296 1.43 KBytes
[ 5] 17.00-18.00 sec 10.9 MBytes 91.5 Mbits/sec 261 1.43 KBytes
[ 5] 18.00-19.00 sec 23.0 MBytes 193 Mbits/sec 444 1.43 KBytes
[ 5] 19.00-20.00 sec 8.86 MBytes 74.3 Mbits/sec 224 1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 224 MBytes 94.1 Mbits/sec 4482
sender
[ 5] 0.00-20.00 sec 224 MBytes 93.9 Mbits/sec
receiver
As suggested before, I have set kern.eventtimer.periodic: 1, the above
tests are done with that setting. I don't see any difference with it set or
not (kern.eventtimer.periodic: 0 was default btw).
There was also a suggestion to have a look at kern.timecouter.*, but I'm
not sure what to look for here:
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 0
kern.timecounter.invariant_tsc: 0
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: XENTIMER(950) ACPI-fast(900) i8254(0) HPET(950)
TSC-low(-100) dummy(-1000000)
kern.timecounter.hardware: XENTIMER
kern.timecounter.alloweddeviation: 5
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.XENTIMER.quality: 950
kern.timecounter.tc.XENTIMER.frequency: 1000000000
kern.timecounter.tc.XENTIMER.counter: 3517394506
kern.timecounter.tc.XENTIMER.mask: 4294967295
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 1749326798
kern.timecounter.tc.ACPI-fast.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 58432
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 62500000
kern.timecounter.tc.HPET.counter: 3606880145
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.TSC-low.quality: -100
kern.timecounter.tc.TSC-low.frequency: 1200075192
kern.timecounter.tc.TSC-low.counter: 3248698599
kern.timecounter.tc.TSC-low.mask: 4294967295
Regards,
Christian
Den ons 26 juni 2019 kl 01:14 skrev Eugene Grosbein <eugen at grosbein.net>:
> 25.06.2019 19:40, Christian M wrote:
>
> > I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig
> xn0
> > -txcsum -rxcsum -lro -tso), and also tried disabling everything for the
> VIF
> > via XCP-ng earlier. It made no difference unfortunately.
>
> Then you should try to bring the interface to promiscuous mode manually
> with "ifconfig xn0 promisc" any retry. If this does not make a difference,
> then you may be facing some timekeeping problem as tcpdump makes additional
> CPU load on the system that can force scheduler behaviour to change.
>
> As next step, you should check your kern.timecounter.* and
> kern.eventtimer.* sysctls.
> If default value of kern.eventtimer.periodic is 0 (it depends on detected
> "hardware"),
> you should change it kern.eventtimer.periodic=1 and retry the test.
>
>
More information about the freebsd-net
mailing list