Unexpected OOM kill on rpi2 while building world

[Mark Millard]

{Sorry for the earlier accidental send before even editing the
text to reply.]

> 
> On 2020-Apr-2, at 16:33, bob prohaska <fbsd at www.zefox.net> wrote:
> 
> Two installations of the 
> FreeBSD-12.1-STABLE-arm-armv7-RPI2-20200305-r358659.img
> image have set up and built -j4  world using a single 64GB
> Samsung Evo Plus microSD card with a 2.6 GB swap partition.
> No changes to /boot/loader.conf required.

The following is from an head -r358966 armv7 example of having
one 3072 MiByte swap/paging partition:

QUOTE
warning: total configured swap (786432 pages) exceeds maximum recommended amount (468832 pages).
warning: increase kern.maxswzone or reduce amount of swap.
END QUOTE

468832 pages is between 1831 MiByte and 1832 MiByte.
2.6 GB is far beyond the recommendation. I've noticed
some variability between armv7 versions for the
recommended figure, but not large differences. So
your context may not be an exact match.

(aarch64 for the same size RAM [1 GiBYte] allows a much
larger swap space without complaint: 3072 MiByte does
not get a complaint on a RPi3 running aarch64 FreeBSD.)

Did you leave things configured such that such a message
was produced on the armv7? What did it say (if produced)?
What was its recommended maximum (translated to, say,
MiBYtes).

If you changed the configuration to avoid the complaint,
what did you change?


Going in a different direction . . .

I'll note that stable/12 -r358659 includes:

stable/12/contrib/googletest/googlemock/test/gmock-matchers_test.cc

which is known to be an issue for OOM activity for
1 GiByte machines, even for -j1 in some configurations.
See:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241848

My experiment showed that built by itself (as if
-j1) on a armv7 with 2 GiByte of RAM it got an
Observed: 1146Mi MaxObs(Act+Wir) . (1740 MiByte
of swap space, but it stayed all free.)

(aarch64 took far more in its analogous experiment.)


> On the third installation, the machine stopped with 
> pid 68521 (c++), jid 0, uid 0, was killed: out of swap space
> so I set 
> vm.pfault_oom_attempts="-1" and restarted buildworld with -DNO_CLEAN
> The machine promptly reported
> pid 93318 (c++), jid 0, uid 0, was killed: out of swap space

(The following is based on head. I've not compared
12-STABLE to be sure how close the match is.)

Possible causes of the OOM kill activity include:

The swap blk uma zone was exhausted.
The swap pctrie uma zone was exhausted.

vm.pfault_oom_attempts and vm.pageout_oom_seq make no
direct difference for these, as far as I know.

Unfortantely, FreeBSD does not specifically report
either cause when it happens, but gives the generic
"out of swap" type of notice.

Your 2.6GB swap space configuration may be making one
or both of these exhaustions more likely. For all I
know "exhaustion" might included something becoming
too fragmented to have individual areas of sufficient
size despite total free in the involved one being
seemingly sufficient.

I'm not claiming that -j4 is even possible to do
reliably, much less staying within the maximum
recommended by default. But, what the consequences
might be for what the warning reports, might put
one outside the generally-well-understood range
of FreeBSD use. Rare expertise might be involved
in understanding what to expect.

> In neither case were there any "indefinite wait...." or any other
> warning messages.

Such messages need not be involved in the uma zone
exhaustions.

> At that point I set
> vm.pageout_oom_seq="4096" and restarted buildworld, again with -DNO_CLEAN.

Which need not contribute to avoiding uma zone
exhaustions.

> Buildworld seems to have gotten past the bottleneck and looks like it
> will complete, but I'm puzzled, both at the appearance of the trouble,
> not formerly seen on the Pi2, and on the failure of
> vm.pfault_oom_attempts="-1" by itself to disable OOMA.

There is no setting that disables all the OOM kill
criteria. The two settings together are not enough
to disable all the OOM kill criteria.

> Casual observation suggests swap use peaks at a few hundred MB
> under 12.1 on the Pi2.

Is 12.1 the version number? The MB figure(s) seem to be missing
from this statement for what is the few hundred MB is under.
Did you mean something like 2.6 GiByte (swap), so fairly near
2.6 GiByte but definitely over 2.0 GiByte of swap in use?

> All three installations were run on the same physical Pi, though
> of course the microSD cards were distinct. Even so, all three 
> cards are nominally identical and likely from the same batch.

buildworld buildkernel would still have lots of variations
in the relative timing of activities during the various
builds. Such could matter to the uma zone usage, for example.
If things are marginal for working, such variable results
might well be expected.

> The one tangible difference is that the Pi2 is now on a private
> network, the two earlier buildworlds were on a public network.
> Can't see how that would matter, however. 


Note:

I have a RPi2B V1.2 with armv7 FreeBSD ( head -r358966 )
doing a -j2 buildworld buildkernel with top watching, top
having my changes that track and report some maximum
observed figures. Maximum Observed swap space usage is
reported, as it MaxObs(Act+Wir).

But it likely will be a day or two before it completes,
presuming it is successful.

(For reference: It is configured like the RPi3 was
for that -j4 test that I reported to you earlier,
other than the swap partition being set to 1800
MiByte and the use of -j2 for armv7.)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)