ZFS eats all memory

[Dr. Nikolaus Klepp]

Hi all!

I have an interesting problem with FreeBSD FreeBSD 12.0-RELEASE-p3 GENERIC. Server is running ZFS mirror with 2 disks, 1TB each, 8GB RAM, 16GB Swap. I had reduced ARC to 4G, but everything ele is default:

vfs.zfs.arc_max=4294967296

All applications under full load use ~ 500MB Ram.

20 days later all memery was used by the kernel, cron could not spawn any subprosses, login and everything despite ZFS was gone - even USB was gone (I tried a USB keyboard, no lights went on). I had to reboot the server and everything went back to normal. The only hint in the logfiles was that cron could not spawn any process.

So I reduced ARC to 2GB:

vfs.zfs.arc_max=2147483648

Warned by the previuse behaviour I closely watched RAM usage. After ~ 7 Days the kernel alread used 7GB RAM, all wired, no swap used. Now I reduced kernel memory and rebooted the machine:

vm.kmem_size=6130489344
vm.kmem_size_max=6130489344

That was yesterday. Now I'm watching wired RAM usage go up again, but I think it does not go above 4GB. top says:

Mem: 98M Active, 190M Inact, 3947M Wired, 3563M Free
ARC: 1968M Total, 979M MFU, 512M MRU, 32K Anon, 85M Header, 392M Other
     1240M Compressed, 3755M Uncompressed, 3.03:1 Ratio
Swap: 16G Total, 16G Free

I am quite surprised by this behaviour. I have not seen this happen on 11.2, but that server had 256GB RAM, not 8GB. 

Is this expected to happen or should I be aware of some other pitfalls? E.g. can I be sure that the system will run longer than 20 days without lockups? 

Nik






-- 
Please do not email me anything that you are not comfortable also sharing with the NSA, CIA ...