Re: ... was killed: a thread waited too long to allocate a page

On 01/02/2024 15:47, Karl Pielorz wrote:

[..]

> I seem to have run into something similar. I recently upgraded a 12.4 
> box to 13.2p9. The box has 32G of RAM, and runs ZFS. We do a lot of 
> rsync work to it monthly - the first month we've done this with 13.2p9 
> we get a lot of processes killed, all with a similar (but not identical) 
> message, e.g.
> 
> pid 11103 (ssh), jid 0, uid 0, was killed: failed to reclaim memory
> pid 10972 (local-unbound), jid 0, uid 59, was killed: failed to reclaim 
> memory
> pid 3223 (snmpd), jid 0, uid 0, was killed: failed to reclaim memory
> pid 3243 (mountd), jid 0, uid 0, was killed: failed to reclaim memory
> pid 3251 (nfsd), jid 0, uid 0, was killed: failed to reclaim memory
> pid 10996 (sshd), jid 0, uid 0, was killed: failed to reclaim memory
> pid 3257 (sendmail), jid 0, uid 0, was killed: failed to reclaim memory
> pid 8562 (csh), jid 0, uid 0, was killed: failed to reclaim memory
> pid 3363 (smartd), jid 0, uid 0, was killed: failed to reclaim memory
> pid 8558 (csh), jid 0, uid 0, was killed: failed to reclaim memory
> pid 3179 (ntpd), jid 0, uid 0, was killed: failed to reclaim memory
> pid 8555 (tcsh), jid 0, uid 1001, was killed: failed to reclaim memory
> pid 3260 (sendmail), jid 0, uid 25, was killed: failed to reclaim memory
> pid 2806 (devd), jid 0, uid 0, was killed: failed to reclaim memory
> pid 3156 (rpcbind), jid 0, uid 0, was killed: failed to reclaim memory
> pid 3252 (nfsd), jid 0, uid 0, was killed: failed to reclaim memory
> pid 3377 (getty), jid 0, uid 0, was killed: failed to reclaim memory
> 
> This 'looks' like 'out of RAM' type situation - but at the time, top 
> showed:

I remember something similar on our machines after upgrade to 13.x about 
a year ago. But don't remember what steps we take to walk around this 
issue (if any). I also see this on my FreeBSD based desktop from time to 
time... "ad more memory" and limit the ARC in loader.conf is my way.

[..]

> I've not seen any bug reports along this line, in fact very little 
> coverage at all of the specific error.
> 
> My only thought is to set a sysctl to limit ZFS ARC usage, i.e. to leave 
> more free RAM floating around the system. During the rsync it was 
> 'swapping' occasionally (few K in, few K out) - but it never ran out of 
> swap that I saw - and it certainly didn't look like an complete out of 
> memory scenario/box (which is what it felt like with everything getting 
> killed).