FYI: Some BPi-M3 notes, including self-hosted buildworld/buildkernel in about 9.5 hours (lldb build included)

[Mark Millard]

I did a from-scratch -j 5 buildworld buildkernel of stable/11 -r308135 on a
BPi-M3 and it took about 9.5 hours. The "Other notes" part below lists the
src-env.conf (src.conf) sort of content, clang's FULL, EXTRAS, and LLDB
included. But WITH_SYSTEM_COMPILER= was in use as well.

The BPi-M3 was running -r308135 that had been cross-built with -mcpu=cortex-a7
listed in XCFLAGS and XCXXFLAGS. The ALLWINNER kernel configuration file was
included in my BPIM3-NODBG and then BPIM3-NODBG made sure that the
performance-tied debug options were disabled in the cross build. (This style
of configuration handling adapts to head's ALLWINNER configuration use as
well --without editing BPIM3-NODBG generally.)

The BPi-M3 is a V1.2 board (with a 1.7mm-inner/4mm-outer power connector) and
has/had:

A) A heat sink on the A83T chip.
B) A 3.3 V fan above the A83T chip.
C) A 240 GByte USB SSD used for / (the root file system) and swap.
   [The SSD is a USB-stick style one, no external power.]
D) There is a 3.1 GByte swap parition on the USB SSD.
   [Given bugzilla 206048 I always avoid using files as swap.]
E) The mmc media used was a Samsung 32GB EVO+.
   [But (C) makes this largely irrelevant to the performance.]
F) A 15W power supply (output: 5V, 3A capable).
G) Appropriate serial port hardware attached.
H) "-mcpu=cortex-a7" listed in XCFLAGS and XCXXFLAGS for
   buildworld/buildkernel.
I) The ALLWINNER kernel configuration file was included in my
   BPIM3-NODBG and then BPIM3-NODBG made sure that the
   performance-tied debug options were disabled. (stable/11
   -r308135)
J) The "Other notes" part below lists the src-env.conf (src.conf)
   sort of content, clang's FULL, EXTRAS, and LLDB included. But
   WITH_SYSTEM_COMPILER= was in use as well.

FreeBSD only supports using 4 of the 8 cores. If 6-8 of them were usable then the
time for a build with some -j from 6 to 9 might be 7 hours or less.

For comparison: As I remember last I did -j 5 buildworld buildkernel of an earlier
stable/11 on a rpi2 it was between 14 hours and 15 hours, no heat sink, no fan,
but an external USB SSD for / (root filesystem) and swap. (That USB SSD on a
powered hub.)

I found the "sysctl -a" aw_ items for the BPi-M3 and the thermal results reported
were 35C or a little less while 4 cores were busy and had been for some time.
Both cluster's reports always agreed when observed --despite only 4 cores being in
use.

(Apparently thermal monitoring and throttling are done by stable/11 for the BPi-M3
--but not via powerd. powerd reports that the cpufreq facilities it uses are are
not available and quits.)



I also used the Ubuntu 16.04 with Mate image and looked around some:

0) All 8 cores (4 from each cluster) are available.

1) The two clusters of 4 cores each are not treated as NUMA from what I can tell.
   [For example the Ubuntu top's NUMA selections report lack of a NUMA context.]

2) (1) is despite the early boot referencing 2 "sockets" [amd64 terminology?] for
   what Allwinner calls "clusters".

3) All the cores are reported as running at the same speed when the speed changes,
   even when only one core was doing much of anything and top shows most as
   0.0% used. This seemed to be true for all the policy settings that I played
   with.

So it appears the Ubuntu image keeps the two clusters of cores set up the same:
when one changes so does the other to match, cores uniform as well. That and
avoiding any assumption of a cache level that spans all the cores (both clusters)
would seem to be the general trend for how things are handled to enable 8 cores.

But that claim is made from the external presentation, not from analysis of
the source code.



Other notes:

I've had the BPi-M3 board change its Ethernet MAC address 3 times so far.
MAC stability may require some explicit control/override somewhere.

I used kern.cam.boot_delay="10000" in /boot/loader.conf so that mounting /
on the USB SSD would reliably go well on the BPi-M3.

I have had the serial connection drop text, unlike the rpi2 using the same
serial hardware and such. I've also had the BPi-M3 context switch to odd
characters being shown.

Crochet tends to set up a file as the swap area. Given bugzilla 206048 I avoid
this configuration on all architectures and create and use a swap partition
instead. (I usually have swap set up and enabled.)

~/src.configs/src.conf.bpim3-clang-bootstrap.bpim3-host was:

> TO_TYPE=armv6
> #
> KERNCONF=BPIM3-NODBG
> TARGET=arm
> .if ${.MAKE.LEVEL} == 0
> TARGET_ARCH=${TO_TYPE}
> .export TARGET_ARCH
> .endif
> #
> #WITH_CROSS_COMPILER=
> WITH_SYSTEM_COMPILER=
> #
> #CPUTYPE=soft
> WITH_LIBCPLUSPLUS=
> WITH_BINUTILS_BOOTSTRAP=
> #WITHOUT_CLANG_BOOTSTRAP=
> WITH_CLANG=
> WITH_CLANG_IS_CC=
> WITH_CLANG_FULL=
> WITH_CLANG_EXTRAS=
> WITH_LLDB=
> #
> WITH_BOOT=
> WITHOUT_LIB32=
> WITHOUT_LIBSOFT=
> #
> WITHOUT_ELFTOOLCHAIN_BOOTSTRAP=
> WITHOUT_GCC_BOOTSTRAP=
> WITHOUT_GCC=
> WITHOUT_GCC_IS_CC=
> WITHOUT_GNUCXX=
> #
> NO_WERROR=
> #WERROR=
> MALLOC_PRODUCTION=
> #
> WITH_DEBUG_FILES=
> #
> XCFLAGS+= -mcpu=cortex-a7
> XCXXFLAGS+= -mcpu=cortex-a7
> # There is no XCPPFLAGS but XCPP gets XCFLAGS content.


===
Mark Millard
markmi at dsl-only.net