Re: FreeBSD 14.0-RELEASE and Raspberry Pi CM4 4GB

From: Mark Millard <marklmi_at_yahoo.com>
Date: Sat, 23 Dec 2023 01:28:04 UTC
On Dec 22, 2023, at 14:48, Mike Karels <mike@karels.net> wrote:

On 22 Dec 2023, at 16:14, Steve Bernacki wrote:
> 
>> Hi Mike,
>> 
>> Indeed, I'm getting a lot of retransmits:
>> 
>> [  5] local 172.16.200.2 port 55551 connected to 172.16.200.182 port 5201
>> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
>> [  5]   0.00-1.00   sec  36.2 MBytes   304 Mbits/sec   60   9.98 KBytes
>> [  5]   1.00-2.00   sec  35.7 MBytes   300 Mbits/sec  143    111 KBytes
>> [  5]   2.00-3.00   sec  34.9 MBytes   293 Mbits/sec  141   7.13 KBytes
>> [  5]   3.00-4.00   sec  33.9 MBytes   284 Mbits/sec  198   99.5 KBytes
>> [  5]   4.00-5.00   sec  34.9 MBytes   292 Mbits/sec  167   1.43 KBytes
>> [  5]   5.00-6.00   sec  34.2 MBytes   287 Mbits/sec  221   2.85 KBytes
>> [  5]   6.00-7.00   sec  34.1 MBytes   286 Mbits/sec  169    100 KBytes
>> [  5]   7.00-8.00   sec  35.2 MBytes   295 Mbits/sec  159   7.13 KBytes
>> [  5]   8.00-9.00   sec  34.3 MBytes   287 Mbits/sec  138   4.28 KBytes
>> [  5]   9.00-10.00  sec  33.3 MBytes   279 Mbits/sec  182   2.85 KBytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval           Transfer     Bitrate         Retr
>> [  5]   0.00-10.00  sec   347 MBytes   291 Mbits/sec 1578             sender
>> [  5]   0.00-10.00  sec   346 MBytes   291 Mbits/sec                  receiver
>> 
>> Thanks,
>> Steve
> 
> One other question: are you running powerd?  I booted without it, and my
> throughput dropped to 600-640 Mb/s.  Repeating the test, retransmissions
> went down but throughput was about the same.  Note, the RPi 4, and probably
> the CM 4, boots at a lower clock frequency by default, and powerd raises it
> under load.  I'm running powerd with -M 1800, overclocking a little.

I explore here fixed frequencies: 2000 MHz, 600 MHz, 1500 MHz, 1800 MHz
(no powerd use)

Based on:

 # uname -apKU
FreeBSD generic 14.0-STABLE FreeBSD 14.0-STABLE #0 stable/14-n266002-2ef9079ece5a: Sat Dec 16 08:49:23 UTC 2023     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 aarch6

# more /boot/efi/config.txt 
[all]
arm_64bit=1
dtparam=audio=on,i2c_arm=on,spi=on
dtoverlay=mmc
dtoverlay=disable-bt
device_tree_address=0x4000
kernel=u-boot.bin

[pi4]
hdmi_safe=1
armstub=armstub8-gic.bin
#
over_voltage=6
sdram_freq_min=3200
arm_freq_min=2000
force_turbo=1

# sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels dev.cpu.0.freq
dev.bcm2835_cpufreq.0.freq_settings: 2000/-1
dev.cpu.0.freq_levels: 2000/-1
dev.cpu.0.freq: 2000


# iperf3 -c 192.168.1.157
Connecting to host 192.168.1.157, port 5201
[  5] local 192.168.1.159 port 52424 connected to 192.168.1.157 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   948 Mbits/sec  243    328 KBytes
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec  150   18.5 KBytes
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec  149    173 KBytes
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec  150    456 KBytes
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec  159    456 KBytes
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec  160    538 KBytes
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec  143   1.43 KBytes
[  5]   7.00-8.00   sec   112 MBytes   942 Mbits/sec  215    167 KBytes
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec  194    580 KBytes
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec  157    552 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec  1720             sender
[  5]   0.00-10.01  sec  1.10 GBytes   941 Mbits/sec                  receiver

iperf Done.


Note: The amd64 system running main [so: 15] and the RPi4B are
on the same ethernet switch.


With the 4 overclocking lines in config.txt commented out :

# sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels dev.cpu.0.freq
dev.bcm2835_cpufreq.0.freq_settings: 1500/-1 600/-1
dev.cpu.0.freq_levels: 1500/-1 600/-1
dev.cpu.0.freq: 600

Note: the default context lacks 1800 (based on the RPi* firmware vintage
in the snapshot). Later I show having 1800 instead of 1500.

# iperf3 -c 192.168.1.157
Connecting to host 192.168.1.157, port 5201
[  5] local 192.168.1.159 port 42060 connected to 192.168.1.157 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  70.8 MBytes   594 Mbits/sec   18    195 KBytes       
[  5]   1.00-2.00   sec  73.8 MBytes   619 Mbits/sec    8    293 KBytes       
[  5]   2.00-3.00   sec  73.6 MBytes   618 Mbits/sec   19    250 KBytes       
[  5]   3.00-4.00   sec  73.6 MBytes   618 Mbits/sec    9    366 KBytes       
[  5]   4.00-5.00   sec  73.3 MBytes   615 Mbits/sec    9    447 KBytes       
[  5]   5.00-6.00   sec  73.3 MBytes   615 Mbits/sec   16    303 KBytes       
[  5]   6.00-7.00   sec  73.2 MBytes   614 Mbits/sec    0    455 KBytes       
[  5]   7.00-8.00   sec  73.6 MBytes   618 Mbits/sec    1    328 KBytes       
[  5]   8.00-9.00   sec  73.5 MBytes   616 Mbits/sec   16    246 KBytes       
[  5]   9.00-10.00  sec  73.3 MBytes   615 Mbits/sec    0    435 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   732 MBytes   614 Mbits/sec   96             sender
[  5]   0.00-10.01  sec   732 MBytes   613 Mbits/sec                  receiver

iperf Done.

Assigning 1500:

# sysctl dev.cpu.0.freq=1500
dev.cpu.0.freq: 600 -> 1500

# sysctl dev.cpu.0.freq=1500
dev.cpu.0.freq: 600 -> 1500
root@generic:~ # iperf3 -c 192.168.1.157
Connecting to host 192.168.1.157, port 5201
[  5] local 192.168.1.159 port 28904 connected to 192.168.1.157 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   949 Mbits/sec    4    472 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec    6    464 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    5    452 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec    3    443 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    4    421 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    4    397 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec    3    378 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec    5    355 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec    2    476 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    5    446 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec   41             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver



Adding arm_boost=1 to config.txt in order to have 1800 instead of 1500
(needed due to the RPi* firmware vintage in FreeBSD snapshots):

# sysctl dev.bcm2835_cpufreq.0.freq_settings dev.cpu.0.freq_levels dev.cpu.0.freq
dev.bcm2835_cpufreq.0.freq_settings: 1800/-1 600/-1
dev.cpu.0.freq_levels: 1800/-1 600/-1
dev.cpu.0.freq: 600

# sysctl dev.cpu.0.freq=1800
dev.cpu.0.freq: 600 -> 1800

# iperf3 -c 192.168.1.157
Connecting to host 192.168.1.157, port 5201
[  5] local 192.168.1.159 port 27499 connected to 192.168.1.157 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   952 Mbits/sec  169    104 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec  150    320 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec  157   52.8 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec  143   87.0 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec  143    121 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec  159    104 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec  138    238 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec  152    276 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec  145    115 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec  162    283 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec  1518             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

iperf Done.



From this it appears that the Retr counts do not seem to make
much of a difference to the Bitrate's achieved. But the arm
frequency does if 600 is involved.


My understanding is that arm_boost=1 was later made the default
in later vintages of the rpi* firmware. arm_boots only causes
1800 for Rev 1.4+ . Pi 400's have 1800 available by default, at
least for modern enough RPi* firmware.

https://www.raspberrypi.com/documentation/computers/config_txt.html
is not necessarily accurate for the older RPi* firmware that FreeBSD
uses in its snapshots/releases.

> IIRC
> the standard clock is 1500 for the RPi 4.  But the throughput is about the
> same using the standard clock with powerd.
> 
> Mike
> 
>> On 12/22/2023 9:23 AM, Mike Karels wrote:
>>> On 22 Dec 2023, at 6:20, Steve Bernacki wrote:
>>> 
>>>> I recently purchased a RPI CM4 with 4GB and 32GB eMMC to replace my aging FreeBSD firewall. I managed to install FreeBSD 14.0-RELEASE-p3 on it, and both Ethernet devices (genet0 and ue0) were properly identified. However, network throughput on my gigabit network is pretty bad; iperf3 reports a maximum transfer speed of 291 Mbits/sec. Flashing OpenWRT on the same hardware using the same ethernet port, I'm able to achieve 923 Mbits/sec.
>>>> 
>>>> Does anyone have any suggestions on how to improve throughput under FreeBSD?
>>>> 
>>>> Thank you
>>>> Steve
>>> I just tested with an RPi4 (4 GB) and 14.0 using iperf3.  It looks like I'm getting
>>> a rather variable number of retransmissions.  On my first run (client on RPi 4),
>>> I got 460 Mb/s with a lot of retransmissions, but the next couple of runs, including
>>> one receiving, I got about 940 Mb even with some retransmissions.  The peers were
>>> fairly fast FreeBSD 13.2 and 15-current systems.  Are you seeing retransmissions?
>>> 
>>> I'll try to look into this, but I'm not sure when I'll get to it.
>>> 
>>> Mike
> 

===
Mark Millard
marklmi at yahoo.com