Re: Can not build kernel on 1GB VM

From: Michael Wayne <freebsd07_at_wayne47.com> 
Date: Fri, 15 Apr 2022 13:49:53 -0400 :

> I have a VM with 1GB RAM running FreeBSD 12.1-RELEASE-p3
> 
> I'm trying to upgrade the machine to 12.3 and having swap failures.
> 
> This machine runs bird to advertise BGP, ssh and not much else so
> the small amount of RAM is (usually) fine.
> 
> For a long time, there was a 1 GB swap file which handled the
> occasional time when excess memory got used.
> 
> Machine needs a custom kernel for BGP, the conf file consists of:
>    include GENERIC
>    ident ROUTING
>    options TCP_SIGNATURE
> 
> 
> Today, while building the 12.3 kernel with:
>    cd /usr/src
>    sudo make toolchain
>    sudo make buildkernel KERNCONF=ROUTING
> the machine ran out of swap. with a bunch of messages like:
>    Apr 15 12:11:26 g1 kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 240593, size: 4096
>    Apr 15 12:11:35 g1 kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 236224, size: 16384
>    Apr 15 12:11:37 g1 kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 245, size: 12288
>    Apr 15 12:11:46 g1 kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 240593, size: 4096
>    Apr 15 12:11:55 g1 kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 236224, size: 16384
>    Apr 15 12:11:57 g1 kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 245, size: 12288
> 
> Thinking it was a sawp space issue, I increased the swap to 4 GB and
> tried again with the same results. Boot gave the kern.maxswzone message,
> I ignored it as I had planned to change as soon as I completed the build.
> 
> So I pulled up top in a console window and watched swap during the
> build.  About 400 MB of RAM was free and about 3 MB of swap was
> used when the machine started linking the kernel:
>    ctfmerge -L VERSION -g -o kernel.full ...
> While this command was running, I saw swap usage go to ~5MB (so
> just over 1%), then started seeing processes being killed due to
> out of swap space.

The "out of swap space" message is usually a misnomer and has
been replaced in main [so: 14], stable/13 , and releng/13.1 :

                case VM_OOM_MEM:
                        reason = "failed to reclaim memory";
                        break;
                case VM_OOM_MEM_PF:
                        reason = "a thread waited too long to allocate a page";
                        break;

(There is one more case that still has the misnomer but
case VM_OOM_SWAPZ seems unlikely to actually happen.)

Given that you are getting the swap_pager: indefinite wait buffer
notices I can not tell which of the two above is happening.

> So, how to proceed?

My /boot/loader/conf has the likes of:

# Delay when persistent low free RAM leads to
# Out Of Memory killing of processes:
vm.pageout_oom_seq=120
#
# For plunty of swap/paging space (will not
# run out), avoid pageout delays leading to
# Out Of Memory killing of processes:
vm.pfault_oom_attempts=-1
#
# For possibly insufficient swap/paging space
# (might run out), increase the pageout delay
# that leads to Out Of Memory killing of
# processes (showing defaults at the time):
#vm.pfault_oom_attempts= 3
#vm.pfault_oom_wait= 10
# (The multiplication is the total but there
# are other potential tradoffs in the factors
# multiplied, even for nearly the same total.)

The vm.pageout_oom_seq=120 delays VM_OOM_MEM.
The vm.pfault_oom_attempts=-1 avoids VM_OOM_MEM_PF.

Note: vm.pfault_oom_attempts=-1 can lead to deadlock
if you actually run out of swap as I understand.

You could try setting both vm.pfault_oom_attempts and
vm.pfault_oom_wait but I've no specific suggested
values for your context.


Note: I do not recommend having so much swap that
you get the the kern.maxswzone message. I do not
recommend adjusting kern.maxswzone as it competes
with other kernel resources --unless you understand
the tradeoffs in fair detail. (I do not understand
them in much detail.)

===
Mark Millard
marklmi at yahoo.com