make buildworld error while compiling 12.0-releng for RPI2

[Mark Millard]

On 2019-Jan-7, at 13:41, Bernd Walter <ticso at cicely7.cicely.de> wrote:

> On Mon, Jan 07, 2019 at 02:59:22PM +0100, Victor wrote:
>> Thank you, Ralf.
>> As a matter of fact I issued 
>> 
>> make buildword
>> 
>> NOT 
>> 
>> make -j4 buildworld
>> 
>> The compilation problem is still there.
> 
> Anyways - this is an out of memory problem.
> dmesg will show you why the process was killed.
> You have to add more swap space.
> Unfortunately compilers get more and more CPU and memory hungry.
> These days I use memory sticks capable of high random load as temporarys
> swap space or NFS space - later mostly for obj and src, not as swap so much.
> For me SanDisk Extreme Go 3.1 and Ultra USB 3.0 worked great.
> That said, I havn't build FreeBSD12 on any ARM myself yet and this is
> only a general experience.
> 

I'm just adding some notes about interpreting the console/dmesg messages,
as one of them is misleading (a misnomer) --plus other related notes.
There was a prolonged list exchange and investigation of buildworld
on rpi2's and rpi3's during 2018, ending in 2018-Sept. The below was
learned during that.


If one is really out of swap space there should be messages like:

Aug  5 17:54:01 sentinel kernel: swap_pager_getswapspace(32): failed

But if one does not see such messages yet does see messages like:

Oct 30 05:15:31 xindi kernel: pid 68935 (rustc), uid 65534, was killed: 
out of swap space

then one is likely not actually out of swap space and adding swap space
will likely not prevent the process kills.

Unfortunately, the wording of that last message is a misnomer for what
drives the kills: it is actually driven by being unable to gain more
free memory but FreeBSD will not swap-out processes that stay runnable
(or are running), only ones that are waiting. Even a single process
that stays runnable and keeps lots of RAM in the active category can
lead to kills when swap is unused or little used. So the kill-behavior
is very workload dependent.



If only the later type of message is showing up, having something like:

# more /etc/sysctl.conf 
vm.pageout_oom_seq=120

with a figure bigger than the default 12, possibly even something like
1024 or 2048 or even more may delay the kills long enough to avoid the
kills because the processes in question then completes first. (The units
are not time but a count, but larger counts take more time before the
process kills start.)


By contrast, if you are seeing the swap_pager_getswapspace messages, you
have to add swap space. There is kern.maxswzone figure that is associated
with swap size and with possible warning pairs when swap space is added,
for example:

warning: total configured swap (524288 pages) exceeds maximum recommended amount (405460 pages).
warning: increase kern.maxswzone or reduce amount of swap.


(An rpi2 V1.1 (armv7) and an rpi3 (aarch64) with the same amount of RAM
get rather different figures at the right.)

There was in stable/12 -r341716 :

MFC r341375 :
Allow to create swap zone larger than v_page_count / 2.

and in stable/11 -r341718 :

MFC r341375 :
Allow to create swap zone larger than v_page_count / 2.

(So release/12.0.0 or release/11.2.0 do not allow such an
increase.)

There is is a kernel memory tradeoff structure to increase in
kern.maxswzone being larger as I understand. Quoting "man 8 loader"
(but the "eight times" is system/architecture specific and will
likely be different):

    kern.maxswzone
                  Limits the amount of KVM to be used to hold swap metadata,
                  which directly governs the maximum amount of swap the
                  system can support, at the rate of approximately 200 MB of
                  swap space per 1 MB of metadata.  This value is specified
                  in bytes of KVA space.  If no value is provided, the system
                  allocates enough memory to handle an amount of swap that
                  corresponds to eight times the amount of physical memory
                  present in the system.

                  Note that swap metadata can be fragmented, which means that
                  the system can run out of space before it reaches the
                  theoretical limit.  Therefore, care should be taken to not
                  configure more swap than approximately half of the
                  theoretical maximum.

                  Running out of space for swap metadata can leave the system
                  in an unrecoverable state.  Therefore, you should only
                  change this parameter if you need to greatly extend the KVM
                  reservation for other resources such as the buffer cache or
                  kern.ipc.nmbclusters.  Modifies kernel option
                  VM_SWZONE_SIZE_MAX.



I also use (not just on small arm hardware):

LDFLAGS.lld+= -Wl,--no-threads

in my equivalents of /etc/make.conf . This avoids lld creating N+2 threads,
N being the the cpu count as FreeBSD counts cpus. This also seems to cut
down on memory use during links.


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)