[Bug 272461] Very low throughput mellanox arm64 Azure
Date: Wed, 12 Jul 2023 05:15:18 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272461
Bug ID: 272461
Summary: Very low throughput mellanox arm64 Azure
Product: Base System
Version: CURRENT
Hardware: arm64
OS: Any
Status: New
Severity: Affects Many People
Priority: ---
Component: kern
Assignee: bugs@FreeBSD.org
Reporter: schakrabarti@microsoft.com
In Azure ARM64 with Mellanox T27800 Family ConnectX-5 Virtual Function, we are
seeing very low throughput of 2.4Gbps instead of available 24Gbps in iperf3
testing. We are using 14.0-CURRENT FreeBSD 14.0-CURRENT.
[root@lisa--447-e0-n1 /home/lisatest]# iperf3 -Vsd
iperf 3.13
FreeBSD lisa--447-e0-n1 14.0-CURRENT FreeBSD 14.0-CURRENT #26
main-n263986-4631191c8a5f-dirty: Thu Jul 6 15:55:36 UTC 2023
root@fbsd13-nvme-test:/data/ws/obj/data/ws/main/arm64.aarch64/sys/GENERIC arm64
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
get_parameters:
{
"tcp": true,
"omit": 0,
"time": 10,
"num": 0,
"blockcount": 0,
"parallel": 1,
"len": 131072,
"pacing_timer": 1000,
"client_version": "3.13"
}
SNDBUF is 32768, expecting 0
RCVBUF is 65536, expecting 0
Time: Wed, 12 Jul 2023 05:12:02 UTC
Accepted connection from 10.0.0.4, port 30147
Cookie: sufbhz7veq7mc2zfkhgfqb3brk7qtekmh35z
TCP MSS: 0 (default)
Congestion algorithm is cubic
[ 5] local 10.0.0.5 port 5201 connected to 10.0.0.4 port 30668
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0
seconds, 10 second test, tos 0
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 1.000051 bytes_transferred 440081080
interval forces keep
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 420 MBytes 3.52 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 0.999994 bytes_transferred 428949692
interval forces keep
[ 5] 1.00-2.00 sec 409 MBytes 3.43 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 1.000187 bytes_transferred 315787268
interval forces keep
[ 5] 2.00-3.00 sec 301 MBytes 2.53 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 1.000033 bytes_transferred 313509888
interval forces keep
[ 5] 3.00-4.00 sec 299 MBytes 2.51 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 0.999786 bytes_transferred 312184546
interval forces keep
[ 5] 4.00-5.00 sec 298 MBytes 2.50 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 1.000079 bytes_transferred 307495692
interval forces keep
[ 5] 5.00-6.00 sec 293 MBytes 2.46 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 1.000039 bytes_transferred 307502682
interval forces keep
[ 5] 6.00-7.00 sec 293 MBytes 2.46 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 0.999980 bytes_transferred 309119866
interval forces keep
[ 5] 7.00-8.00 sec 295 MBytes 2.47 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 0.999948 bytes_transferred 305474486
interval forces keep
[ 5] 8.00-9.00 sec 291 MBytes 2.44 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 1.000009 bytes_transferred 305829276
interval forces keep
[ 5] 9.00-10.00 sec 292 MBytes 2.45 Gbits/sec
tcpi_snd_cwnd 13980 tcpi_snd_mss 1410 tcpi_rtt 10000
interval_len 0.000524 bytes_transferred 0
ignoring short interval with no data
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate
[ 5] (sender statistics not available)
[ 5] 0.00-10.00 sec 3.12 GBytes 2.68 Gbits/sec receiver
rcv_tcp_congestion cubic
get_results
{
"cpu_util_total": 71.699362492382051,
"cpu_util_user": 0.1566976289611289,
"cpu_util_system": 71.589782122913959,
"sender_has_retransmits": 1,
"congestion_used": "newreno",
"streams": [{
"id": 1,
"bytes": 3347795132,
"retransmits": 109,
"jitter": 0,
"errors": 0,
"packets": 0,
"start_time": 0,
"end_time": 10.000558
}]
}
send_results
{
"cpu_util_total": 35.778942258224347,
"cpu_util_user": 0.15585994703341635,
"cpu_util_system": 35.731875294043533,
"sender_has_retransmits": 18446744073709551615,
"congestion_used": "cubic",
"streams": [{
"id": 1,
"bytes": 3345934476,
"retransmits": 18446744073709551615,
"jitter": 0,
"errors": 0,
"packets": 0,
"start_time": 0,
"end_time": 10.00063
}]
}
iperf 3.13
Also client side is showing retransmit and then speed drops:
# iperf3 -c 10.0.0.5
Connecting to host 10.0.0.5, port 5201
[ 5] local 10.0.0.4 port 30668 connected to 10.0.0.5 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 421 MBytes 3.52 Gbits/sec 0 1.61 MBytes
[ 5] 1.00-2.00 sec 410 MBytes 3.44 Gbits/sec 107 579 KBytes
[ 5] 2.00-3.00 sec 302 MBytes 2.53 Gbits/sec 0 879 KBytes
[ 5] 3.00-4.00 sec 299 MBytes 2.51 Gbits/sec 0 886 KBytes
[ 5] 4.00-5.00 sec 297 MBytes 2.50 Gbits/sec 0 910 KBytes
[ 5] 5.00-6.00 sec 293 MBytes 2.46 Gbits/sec 0 961 KBytes
[ 5] 6.00-7.00 sec 293 MBytes 2.46 Gbits/sec 0 982 KBytes
[ 5] 7.00-8.00 sec 295 MBytes 2.47 Gbits/sec 2 884 KBytes
[ 5] 8.00-9.00 sec 291 MBytes 2.45 Gbits/sec 0 929 KBytes
[ 5] 9.00-10.00 sec 292 MBytes 2.45 Gbits/sec 0 973 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.12 GBytes 2.68 Gbits/sec 109 sender
[ 5] 0.00-10.00 sec 3.12 GBytes 2.68 Gbits/sec receiver
ifconfig output:
root@schakrabarti-freebsd:/datadrive/sandbox_21_06/src/sys # ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
hn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8051b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,LRO,LINKSTATE>
ether 60:45:bd:d4:97:44
inet 172.30.0.4 netmask 0xffffff00 broadcast 172.30.0.255
media: Ethernet 50GBase-KR4 <full-duplex,rxpause,txpause>
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
mce0: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=8805bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,LINKSTATE>
ether 60:45:bd:d4:97:44
media: Ethernet 50GBase-KR4 <full-duplex,rxpause,txpause>
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Please help me to solve this issue as it will block FreeBSD on Azure ARM64.
--
You are receiving this mail because:
You are the assignee for the bug.