Re: ... was killed: a thread waited too long to allocate a page [actually: was killed: failed to reclaim memory problem]

On Feb 1, 2024, at 10:31, Karl Pielorz <kpielorz_lst@tdx.co.uk> wrote:

> --On 01 February 2024 08:30 -0800 Mark Millard <marklmi@yahoo.com> wrote:
>> [snip]
>> 
>> One direction of control is . . .
>> 
>> What do you have for ( copied from my /boot/loader.conf ):
>> 
>> #
>> # Delay when persistent low free RAM leads to
>> # Out Of Memory killing of processes:
>> vm.pageout_oom_seq=120
> 
> loader.conf is empty - it's a pretty stock / out the box install.
> 
>> The default is 12 (last I knew, anyway).
> 
> Yes, default is 12 here.
> 
>> The 120 figure has allowed me and others to do buildworld,
>> buildkernel, and poudriere bulk runs on small arm boards
>> using all cores that otherwise got "failed to reclaim
>> memory" (to use the modern, improved [not misleading]
>> message text). Similarly for others that had other kinds
>> of contexts that got the message.
>> 
>> (The units for the 120 are not time units: more like a
>> number of (re)tries to gain at least a target amount of
>> Free RAM before failure handling starts. The comment
>> wording is based on a consequence of the assignment.)
>> 
>> The 120 is not a maximum, just a figure that has proved
>> useful in various contexts.
>> 
>> But see the notes below based as well.
> 
> Thanks for the notes. I think I can reproduce this (but it's painful to do - as is pretty much any production server killing all it's procs) so I'll wait until next run. I've set the vm.pageout_oom_seq=120 if only because it's a runtime option (no reboot).

Yep: sysctl -T vm.pageout_oom_seq shows it (so: a tunable)
and sysctl -W vm.pageout_oom_seq shows it (so: a writable).

> It's been many, many years since I had to do anything with default arc sizing on FreeBSD (thankfully) - something I wouldn't want to really do again for such a 'simple' machine. I've noted the change here - and we'll see how it goes.
> 
>> That 24G+ of wired meant that only 8GiBytes- were
>> available everything else. Avoiding that by limiting
>> the ARC (tuning ZFS) or adjusting how the work load
>> is spread over time or some combination also looks
>> appropriate.
> 
> tbh - 8GiB is probably more than enough for what's running on that machine (very little) - but I take your point.
> 

That you got the kill's indicates otherwise for
the before-kills context: the free RAM had to be
much smaller than that to get the kills (see notes
below). Getting information from the time
just before the kills start, such as top output,
could be very interesting. Unfortunately, getting
such tends to be messy.

But the likes of vm.pageout_oom_seq=120 helps with
the issue of observerving the bad conditions because
they last longer. Even if vm.pageout_oom_seq=120
prevents the kills, you should be able to observe
how bad things get during the bad time. (Presuming
you can be there to observe in the proper time
frame.)


Of interest would also be the likes of:

# sysctl -d vm.stats.vm | grep v_free
vm.stats.vm.v_free_severe: Severe page depletion point
vm.stats.vm.v_free_count: Free pages
vm.stats.vm.v_free_min: Minimum low-free-pages threshold
vm.stats.vm.v_free_target: Pages desired free
vm.stats.vm.v_free_reserved: Pages reserved for deadlock

So (not from your bad context):

# sysctl vm.stats.vm | grep v_free
vm.stats.vm.v_free_severe: 30540
vm.stats.vm.v_free_count: 6072295
vm.stats.vm.v_free_min: 50578
vm.stats.vm.v_free_target: 170806
vm.stats.vm.v_free_reserved: 10502

This was from an aarch64 context with:

real memory  = 33698009088 (32136 MB)
avail memory = 32827478016 (31306 MB)

In decreasing order (ignoring the live
count):

vm.stats.vm.v_free_target: 170806
vm.stats.vm.v_free_min: 50578
vm.stats.vm.v_free_severe: 30540
vm.stats.vm.v_free_reserved: 10502

You may want to look those up on your system.

FYI:

170806 * 4096 Bytes/page is somewhat over
667 MiBytes.

50578 * 4096 Bytes/page is somewhat under
198 MiBytes.

30540 * 4096 Bytes/page is somewhat over
119 MiBytes.

===
Mark Millard
marklmi at yahoo.com