ZFS memory use (arc limit)

[Matt Churchyard]

Hello,

What's the current best practices for handling memory use for ZFS ARC?

I've been a ZFS user for over 10 years starting with v15 pretty much back when it first appeared.
Very quickly I adopted the attitude of immediately limiting ARC manually via sysctl, as I seemed to get guaranteed swap issues otherwise. Over the years I've seen references to other people seeing the same issues, various work on ZFS memory use, but I've never relied on the system to sort itself out and always just hard limited it by default.

Just this week I installed a new system (11.2-REL) with 12GB RAM (not a huge amount but more than most my FreeBSD systems) that does nothing else other than NFS sharing. I never got round to limiting the ARC and thought it would probably be alright considering how mature the ZFS support is these days. I did check the sysctl and it seemed to be set around 11GB or so. Just over 24 hours after booting the NFS shares went down and a check of dmesg shows the familiar "x was killed; out of swap space" errors. Swap space is 4GB, which could probably be higher, but I've never seen it above a few MB in use. (Maybe I need to set it the same as RAM?)

As such, it seems that it's still absolutely vital that the admin sets a hard limit on ARC, leaving enough headroom for whatever applications are being run on the system?
Is this recommended, or is there something I'm not doing right...?

Regards,
Matt Churchyard