From nobody Sat Dec 23 01:28:04 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sxml65Qdpz54lRt for ; Sat, 23 Dec 2023 01:28:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-25.consmr.mail.gq1.yahoo.com (sonic303-25.consmr.mail.gq1.yahoo.com [98.137.64.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sxml62VNvz4rml for ; Sat, 23 Dec 2023 01:28:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703294896; bh=q9u2pl7FTJEt4M2jFHz3xk0jPR/FRA7L2JG3GgOstvE=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Hmv+TrF9bKn7gIoGv3tURgO7DCvNgLQa9fzjl/nLfwM43LUkz0mv/MonVx5vN6Hc+VZr13MC7BtrFXmXFV+ZRP5Lm3ljzedKNgc0dUbPuJRp1Fkkz76I5OPTl7ta2DqOtmIiBnUHxrdwVF7b+pDaQfgWenodK+rnPJeCY/50fcGd3hwlhp31iajaVP6lIxcePDKujoZdEdj+47wWXTsovTL+Majq/YUb1Kv5bjRR/sS1gOsX4uWL6FLD91kU02hzifWFkxau896Oc3Z4nHuyMlAH9Ghi9AcatCOJFPUbF/Lx+5OODG0ZWBd0+5ZsHwGhqOio00Wx8D2T08U3wh+LlA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703294896; bh=v/ppSP8oNMgM+cJRTXWFd5gPWF9uAB2KcbTXO0MV0ux=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=seUQrilvSi6cRZhz0XPmWOUNpTHgGZtDmL5aGyv7jJxW5WViC8qT1TgwOev4WijyOEqEgzEzdBxydFFiuoakmK9hic5zeBzWTkailByP81goEYIMSU3NiXx9yQi35WCDEUKWetDbKvg9f2M7z2+GFmRBV8aViqifRru3xoE86aivAL/508uG+hZxz+xKlzvL/hy0/GNpqd8J7ziQYxJ6pxa1sYHb5acVJKSefu72tD12azbiwjz5hRLmHRcaXy4wm4ggFIKr12KLQf0N+F+WRbNxCCVsXIaFjN4II0AhfNAVFzp9gqboh0h6GwH3Nd9lSr/ujGACrrvddI4i/+HuqQ== X-YMail-OSG: u.YJOXoVM1lj0eK9Wkh7M32.Jhd08UanT.ac.975_fF.NBOwDoOsuw7cDuFc9_w wCLHpb2GrgpDr_gPFFNryRy5HBItj9XotjXPS1JhrJ4q9u4pqD5.MSA71BV5IVi.fF21vw5VAfeI g3GJp10bFTJPvsDoKgS0Uzbl1dODZ1Wj6GGaD3wEjGb7tCb5BKqd1eE7MjbKz12dIGDDqzzDEEQP xf9Q1J0nSR0Po3iVSgpB.U1hzPFcsrPd6tlQrt4VdmmZQtmLaD_nYlssfHo2TV.i.Zx1n97zNA_4 9ccD._vhWmdpELbJAdymA3gVMBnNpVRkF1erpFueAzBAhARDRI5xMKFsFcFD_r6v_1P50dm7ytcB 0azpiw8UCh2pR6Z08fojor.VVgDs_Aw.DAP2j1VG9gchKNsle1l.wdfxIk9oILBb.VmJ1VrLcvwJ xuZDyBu3zMYSN09EWhPLLZ.r6s5NTv9jQYX87mL_26qRvxZkwjGrhqQF4CDy8DLi75Qd52.033MA kmsBm2XYHvUGzLucsqcqyD1Oyu.od1aJP.w3y7LiNuCgOslHzO5ACeH9kwm.wN1nCVya3h_hGBwr O_UlK4q4BrF628vyAg2.mgCGIL0TI7_z0MQq1EycTJksB3pWynsrOUt2t6jz2LI93YhvQLC1Uvaz FioUS0PmQHRqwqRAKVjXLfD9JsxfPhdl7JvGYsKH0hAaecmp82VnnDYF1sFghqMjhlqF2Iei41CD Vdfawhf9lLNt6FDnp7zzTtuVtjhu3leFCjWauIVIvTHHouPBbnd4E5ShWq3yx_4vg1myrimVIeIc ZORL7hOIn03bKxhJqXA55fWE42u_3MIZL_8OjUnjd7mRLUVHuLTbjVWB2bC91yRQV2IWow.yTLa2 udv198XynnyQtOzXXUn.z3R4RkszM5jRFMGGnG8xV4PhA3KefM9nm0nm1osPzVEiin3qcocsoyB5 hjWomVnjgF5PMwgyK6a5ZqviX_iEU_af4XG.n5nZpHs4BT8wl2MoX86u8dRZHDsgajmVLNaVny_e Yy.8IF5JMCroY6xsej7ivlN3jD4L61XgVx_dgsfuMxyhObXW5wGGI1_jY7awve4iDuxKfMql_2Yp tMI4mZQUQSEBzUzfibvlTwY78NFEqN_C0CWykf2aTuq51enVb.Hc23TRe1B8cD8f602k_ScPEBJk llxrk7PPjFbhvWpO2KAfYaHeo3LUPe.AeWArmWMYIzBRj6gxCxlB_zdBsFnupwCwRZZ5p63ZpbMI MSEsi3Sp6WqmdcnzGEQUitd0U65CeizlLGmrwqLuDj0QnmVmcYUZqLUk8wsCiDm4wmenLr4ssZQd 3WWGLUV43mVMCHrU1V06N2n.UnLObHU47aDG3mz.DloDiE9TePLhygj05G6dEPRUmd3mIPDmojfx 1Hp7DjnAdafY5Wi5NoYpyVGnJt1sWFdNqabyC7gf_CTY8E7LnBb6pPXMIxJ7F0MeT6tBGX0qtYkc rsF0F88XsSMWnic4QE9KFL.kyiaK9__QwuJlM01SlrfSu_Rc2OXmrbJ85EQHi.Fn1DKvusCj5AHk PmeVCG5fU_u4oc6tzYocHvXA3bm8XCJp8lUOC76ldh_XvT.KhkiqKXV_4Ts4NEmkXB3JFUDiMadZ X3t8xw_gP1TgQ7MknTdEajE5EcB.4HH9rJ597NmVwCsaPM23KaER6aDhWfy235tWQ_XAUUuRmnuX FKcctT0rcDUYnrJfzdCDJZBsLNMGuqGZ5ZyNLQiskOPi3_sjbCYiARtwf9OBuK_OMHy3FE3R6vbI nvhtA83LoRr5GML8mpdxzdWCq87SG_dza9Npy1HqIMkwghiCAknjWOdfujYT.ozf2e40zbPrlmNz 7WytHyfk3bwohCaeuSsyRxstLFuFima3FbMruQCgEXwqEXMTNlTZ93Vb.VFA6oNF3eG7R_DVNqsW pych.ROKeoaJBae7y6RJN9Zza_HtJIQz0n3xzaARNqsptF0gI09Zws0FizQUyzTfDGXgXP9Q8lvv Sn3FvP9lcfGcPZCQ4Dwc6X2OA378MUDs9Q7z3Q6icmyOjR9OG.QCGCyKSLmAkcD11RPLqXzm3EnC KPComfBtFKyC7FefVGI5v3ikcAP5Q4JkYfVgIA9bjtawcLK4GYL7_yCa4hZXke9iaILkQ3crykNK yhx4Xlm0Uxm.CgxZzCe_UMq98BJuPLPKQXIgQLy94gIsa1dZ7LboSGC5IltyDrmZ.YF9WZWr6Opl ThmFuzdlVWAaKOjgoXfQ90OvPBtczPql1Mjgx3OUHC6iUbraNsquE6JdPFMfFFvOmakjl7EEvOb. h X-Sonic-MF: X-Sonic-ID: 248504ce-7509-4f5a-a592-67c23e9d22be Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Sat, 23 Dec 2023 01:28:16 +0000 Received: by hermes--production-gq1-6949d6d8f9-k52jv (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 26a369e05ba9cb26f9c6b47ac4976467; Sat, 23 Dec 2023 01:28:15 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: FreeBSD 14.0-RELEASE and Raspberry Pi CM4 4GB From: Mark Millard In-Reply-To: <2E1B887B-EB3C-4F47-A6EE-8256149F7C84@karels.net> Date: Fri, 22 Dec 2023 17:28:04 -0800 Cc: Steve Bernacki , "freebsd-arm@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <445940f7-e8f1-4dbc-87be-99bfd705141d@copacetic.net> <2E1B887B-EB3C-4F47-A6EE-8256149F7C84@karels.net> To: Mike Karels X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Sxml62VNvz4rml On Dec 22, 2023, at 14:48, Mike Karels wrote: On 22 Dec 2023, at 16:14, Steve Bernacki wrote: >=20 >> Hi Mike, >>=20 >> Indeed, I'm getting a lot of retransmits: >>=20 >> [ 5] local 172.16.200.2 port 55551 connected to 172.16.200.182 port = 5201 >> [ ID] Interval Transfer Bitrate Retr Cwnd >> [ 5] 0.00-1.00 sec 36.2 MBytes 304 Mbits/sec 60 9.98 = KBytes >> [ 5] 1.00-2.00 sec 35.7 MBytes 300 Mbits/sec 143 111 = KBytes >> [ 5] 2.00-3.00 sec 34.9 MBytes 293 Mbits/sec 141 7.13 = KBytes >> [ 5] 3.00-4.00 sec 33.9 MBytes 284 Mbits/sec 198 99.5 = KBytes >> [ 5] 4.00-5.00 sec 34.9 MBytes 292 Mbits/sec 167 1.43 = KBytes >> [ 5] 5.00-6.00 sec 34.2 MBytes 287 Mbits/sec 221 2.85 = KBytes >> [ 5] 6.00-7.00 sec 34.1 MBytes 286 Mbits/sec 169 100 = KBytes >> [ 5] 7.00-8.00 sec 35.2 MBytes 295 Mbits/sec 159 7.13 = KBytes >> [ 5] 8.00-9.00 sec 34.3 MBytes 287 Mbits/sec 138 4.28 = KBytes >> [ 5] 9.00-10.00 sec 33.3 MBytes 279 Mbits/sec 182 2.85 = KBytes >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.00 sec 347 MBytes 291 Mbits/sec 1578 = sender >> [ 5] 0.00-10.00 sec 346 MBytes 291 Mbits/sec = receiver >>=20 >> Thanks, >> Steve >=20 > One other question: are you running powerd? I booted without it, and = my > throughput dropped to 600-640 Mb/s. Repeating the test, = retransmissions > went down but throughput was about the same. Note, the RPi 4, and = probably > the CM 4, boots at a lower clock frequency by default, and powerd = raises it > under load. I'm running powerd with -M 1800, overclocking a little. I explore here fixed frequencies: 2000 MHz, 600 MHz, 1500 MHz, 1800 MHz (no powerd use) Based on: # uname -apKU FreeBSD generic 14.0-STABLE FreeBSD 14.0-STABLE #0 = stable/14-n266002-2ef9079ece5a: Sat Dec 16 08:49:23 UTC 2023 = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC = arm64 aarch6 # more /boot/efi/config.txt=20 [all] arm_64bit=3D1 dtparam=3Daudio=3Don,i2c_arm=3Don,spi=3Don dtoverlay=3Dmmc dtoverlay=3Ddisable-bt device_tree_address=3D0x4000 kernel=3Du-boot.bin [pi4] hdmi_safe=3D1 armstub=3Darmstub8-gic.bin # over_voltage=3D6 sdram_freq_min=3D3200 arm_freq_min=3D2000 force_turbo=3D1 # sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels = dev.cpu.0.freq dev.bcm2835_cpufreq.0.freq_settings: 2000/-1 dev.cpu.0.freq_levels: 2000/-1 dev.cpu.0.freq: 2000 # iperf3 -c 192.168.1.157 Connecting to host 192.168.1.157, port 5201 [ 5] local 192.168.1.159 port 52424 connected to 192.168.1.157 port = 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 243 328 KBytes [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 150 18.5 KBytes [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 149 173 KBytes [ 5] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 150 456 KBytes [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 159 456 KBytes [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 160 538 KBytes [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 143 1.43 KBytes [ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 215 167 KBytes [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 194 580 KBytes [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 157 552 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 1720 = sender [ 5] 0.00-10.01 sec 1.10 GBytes 941 Mbits/sec = receiver iperf Done. Note: The amd64 system running main [so: 15] and the RPi4B are on the same ethernet switch. With the 4 overclocking lines in config.txt commented out : # sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels = dev.cpu.0.freq dev.bcm2835_cpufreq.0.freq_settings: 1500/-1 600/-1 dev.cpu.0.freq_levels: 1500/-1 600/-1 dev.cpu.0.freq: 600 Note: the default context lacks 1800 (based on the RPi* firmware vintage in the snapshot). Later I show having 1800 instead of 1500. # iperf3 -c 192.168.1.157 Connecting to host 192.168.1.157, port 5201 [ 5] local 192.168.1.159 port 42060 connected to 192.168.1.157 port = 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 70.8 MBytes 594 Mbits/sec 18 195 KBytes = =20 [ 5] 1.00-2.00 sec 73.8 MBytes 619 Mbits/sec 8 293 KBytes = =20 [ 5] 2.00-3.00 sec 73.6 MBytes 618 Mbits/sec 19 250 KBytes = =20 [ 5] 3.00-4.00 sec 73.6 MBytes 618 Mbits/sec 9 366 KBytes = =20 [ 5] 4.00-5.00 sec 73.3 MBytes 615 Mbits/sec 9 447 KBytes = =20 [ 5] 5.00-6.00 sec 73.3 MBytes 615 Mbits/sec 16 303 KBytes = =20 [ 5] 6.00-7.00 sec 73.2 MBytes 614 Mbits/sec 0 455 KBytes = =20 [ 5] 7.00-8.00 sec 73.6 MBytes 618 Mbits/sec 1 328 KBytes = =20 [ 5] 8.00-9.00 sec 73.5 MBytes 616 Mbits/sec 16 246 KBytes = =20 [ 5] 9.00-10.00 sec 73.3 MBytes 615 Mbits/sec 0 435 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 732 MBytes 614 Mbits/sec 96 = sender [ 5] 0.00-10.01 sec 732 MBytes 613 Mbits/sec = receiver iperf Done. Assigning 1500: # sysctl dev.cpu.0.freq=3D1500 dev.cpu.0.freq: 600 -> 1500 # sysctl dev.cpu.0.freq=3D1500 dev.cpu.0.freq: 600 -> 1500 root@generic:~ # iperf3 -c 192.168.1.157 Connecting to host 192.168.1.157, port 5201 [ 5] local 192.168.1.159 port 28904 connected to 192.168.1.157 port = 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 949 Mbits/sec 4 472 KBytes = =20 [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 6 464 KBytes = =20 [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 5 452 KBytes = =20 [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 3 443 KBytes = =20 [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 4 421 KBytes = =20 [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 4 397 KBytes = =20 [ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 3 378 KBytes = =20 [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 5 355 KBytes = =20 [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 2 476 KBytes = =20 [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 5 446 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 41 = sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec = receiver Adding arm_boost=3D1 to config.txt in order to have 1800 instead of 1500 (needed due to the RPi* firmware vintage in FreeBSD snapshots): # sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels = dev.cpu.0.freq dev.bcm2835_cpufreq.0.freq_settings: 1800/-1 600/-1 dev.cpu.0.freq_levels: 1800/-1 600/-1 dev.cpu.0.freq: 600 # sysctl dev.cpu.0.freq=3D1800 dev.cpu.0.freq: 600 -> 1800 # iperf3 -c 192.168.1.157 Connecting to host 192.168.1.157, port 5201 [ 5] local 192.168.1.159 port 27499 connected to 192.168.1.157 port = 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 169 104 KBytes = =20 [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 150 320 KBytes = =20 [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 157 52.8 KBytes = =20 [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 143 87.0 KBytes = =20 [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 143 121 KBytes = =20 [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 159 104 KBytes = =20 [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 138 238 KBytes = =20 [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 152 276 KBytes = =20 [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 145 115 KBytes = =20 [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 162 283 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 1518 = sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec = receiver iperf Done. =46rom this it appears that the Retr counts do not seem to make much of a difference to the Bitrate's achieved. But the arm frequency does if 600 is involved. My understanding is that arm_boost=3D1 was later made the default in later vintages of the rpi* firmware. arm_boots only causes 1800 for Rev 1.4+ . Pi 400's have 1800 available by default, at least for modern enough RPi* firmware. https://www.raspberrypi.com/documentation/computers/config_txt.html is not necessarily accurate for the older RPi* firmware that FreeBSD uses in its snapshots/releases. > IIRC > the standard clock is 1500 for the RPi 4. But the throughput is about = the > same using the standard clock with powerd. >=20 > Mike >=20 >> On 12/22/2023 9:23 AM, Mike Karels wrote: >>> On 22 Dec 2023, at 6:20, Steve Bernacki wrote: >>>=20 >>>> I recently purchased a RPI CM4 with 4GB and 32GB eMMC to replace my = aging FreeBSD firewall. I managed to install FreeBSD 14.0-RELEASE-p3 on = it, and both Ethernet devices (genet0 and ue0) were properly identified. = However, network throughput on my gigabit network is pretty bad; iperf3 = reports a maximum transfer speed of 291 Mbits/sec. Flashing OpenWRT on = the same hardware using the same ethernet port, I'm able to achieve 923 = Mbits/sec. >>>>=20 >>>> Does anyone have any suggestions on how to improve throughput under = FreeBSD? >>>>=20 >>>> Thank you >>>> Steve >>> I just tested with an RPi4 (4 GB) and 14.0 using iperf3. It looks = like I'm getting >>> a rather variable number of retransmissions. On my first run = (client on RPi 4), >>> I got 460 Mb/s with a lot of retransmissions, but the next couple of = runs, including >>> one receiving, I got about 940 Mb even with some retransmissions. = The peers were >>> fairly fast FreeBSD 13.2 and 15-current systems. Are you seeing = retransmissions? >>>=20 >>> I'll try to look into this, but I'm not sure when I'll get to it. >>>=20 >>> Mike >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com