spurious out of swap kills

[John Kennedy]

On Fri, Sep 13, 2019 at 09:33:02PM +1000, Trev wrote:
> [From the FreeBSD-Current mailing list]
>
> Konstantin Belousov wrote on 13/09/2019 15:53:
> > Basically, page fault handler waits for vm.pfault_oom_wait *
> > vm.pfault_oom_attempts for a page allocation before killing the process.
> > Default is 30 secs, and if you cannot get a page for 30 secs, there is
> > something very wrong with the machine.
>
> This may well explain the oom kills that were being experienced on 
> several Raspberry Pi 3B+ boards using the microSD as swap and why 
> upgrading the microSD to the fastest available or using a USB attached 
> hard drive did not suffer from the issue.

  I might argue about "very wrong", unless you include humans trying to push
the bounds of what they should be doing.  (:

  I stress the hell out of my RPI3B+.  Like poudraire jail builds on top of ZFS.
Compiling llvm80 or rust with anything else is almost guaranteed to be a
Truxican standoff.  But I'm convinced FreeBSD is trying to do the right thing,
but that's a LOT of random workload elements that swapping in and out all the
time.  It's a relatively tiny amount of RAM.  In my experience, I haven't seen
OOM from "slow"-swap, I see "indefinite wait buffer" errors (which don't seem
to be fatal in and of themselves, but will manifest as paused jobs:

	Sep 11 07:12:50 rpi3 kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 135598, size: 40960

  I still run out of swap space too of course.  Mind you, I've allocated ~8G:

	Sep 11 07:12:50 rpi3 kernel: pid 42342 (python2.7), jid 9, uid 65534, was killed: out of swap space

  For my RPI3B+, I initially used an older card I had from my RPI1 and I used
the recipe that said to create swap-on-filesystem.  Both of those were terrible
IMHO.  I upgraded to the best sanely-priced card my local Best Buy had and made
sure it was movie-rated (since I figured streaming video to disk was a good
analog for disk I/O).  I also moved the swap to raw partition-on-"disk" to
avoid unnecessary extra buffering.  Ultimately I added a USB disk as well.

  Like you, I basically decided that giving the I/O as much pony-power as I
could helped it to power out of bad situations, but I also have to acknowledge
that I'm regularly creating unreasonably bad situations.  (: