ZFS overgrowing arc_max -> OOM kills

[Peter]

It seems this issue is a combination of two different matters.

On Sun, 01 Dec 2019 12:48:29 +0100, Peter Much  
<pmc at citylink.dinoex.sub.org> wrote:

>> Kernel Memory Map:                              1.38    GiB
>>         Size:                           98.79%  1.36    GiB
>>         Free:                           1.21%   17.10   MiB

Now I worked myself thru the code in arc.c, and everything there appears  
to work correctly.

What I seem to have is about 85% metadata, and 75% of what is called  
"overhead" data (arcstats.overhead_size), and that seems not evictable. So  
in that case the ARC goes beyond it's upper limit, and it goes beyond the  
provisioned kmem just the same way, because it's the same code. So, the  
exitus is inevitable.

The other issue is the OOM kills, and that is the same issue as already  
reported by many people (bug 232800 and 231457, where Mark Millard  
declared this behaviour "works as designed"). It is probably unrelated to  
ZFS (except that ZFS does not tend to improve memory abundance).

So now I go and hack my arc.c, and probably hack the vm pageout as well,  
and see how to make that stuff behave.
(Commentary and inspiration is welcome, as usual.)

rgds,
PMc