All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

[Mark Martinec]

2018-08-13 21:48, Volodymyr Kostyrko wrote:
> I've been in the same situation. ZFS, only pool, no ZFS errors.
> 
> I think the problem is rather between swapping and ZFS ARC. This host
> has different load, sometimes it needs more active memory, somtimes
> less... This means that active zone can expand and shrink like +-2G os
> mem (I have 16Gb installed there). The problem is, when huge task is
> idle it doesn't use much active memory and other activity is pushing
> it's memory to the swap. When active runs low and ARC runs >50% of
> memory it becomes very hard to make ARC give some memory back. My host
> even was broght to the point when it couldn't get tasks back into
> memory from swap, because while some pages were restored from swap the
> time passes by and the other pages are instead stored to swap due to
> zome ARC activity. Finally active zone shrinks so bad that the host
> becomes unresponsive.
> 
> Like 6 month ago I tried tweaking kernel and swap to make things go
> other way. Currently I have `vm.swap_idle_enabled=1` in
> /etc/loader.conf and looks like this solves my problem. The other
> interesting things to look at are `vfs.zfs.arc_free_target`,
> `vfs.zfs.arc_shrink_shift`, `vfs.zfs.arc_grow_retry`.
> 
> Or you can take another route and plain limit current ARC size with
> `vfs.zfs.arc_max`.

What you describe is not the same problem as the one I described
in this thread. In my case the ZFS malloc'ed memory ("solaris" zone)
is growing, while the size of the ARC remains capped to a reasonably
low value, and the ARC even shrinks as the "solaris" zone approaches
the memory size.

I too have been bitten previously by the ARC size being reluctant to
shrink. Ths problem is described here, but only partially mitigated
now in the 11.? version:

   https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594

The usually suggested workaround is to limit the size of the ARC,
although it would be nice to find a solution to handle ARC UMA
shrinking automatically, like it worked well in FreeBSD 9 but
broke in FreeBSD 10.

Like I said, the problem I described in this thread is different.

   Mark