Strange ARC/Swap/CPU on yesterday's -CURRENT

[Trond Endrestøl]

On Mon, 5 Mar 2018 14:39-0600, Larry Rosenman wrote:

> Upgraded to:
> 
> FreeBSD borg.lerctr.org 12.0-CURRENT FreeBSD 12.0-CURRENT #11 r330385: Sun Mar  4 12:48:52 CST 2018     root at borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/VT-LER  amd64
> +1200060 1200060
> 
> Yesterday, and I'm seeing really strange slowness, ARC use, and SWAP use and swapping.
> 
> See http://www.lerctr.org/~ler/FreeBSD/Swapuse.png

I see these symptoms on stable/11. One of my servers has 32 GiB of 
RAM. After a reboot all is well. ARC starts to fill up, and I still 
have more than half of the memory available for user processes.

After running the periodic jobs at night, the amount of wired memory 
goes sky high. /etc/periodic/weekly/310.locate is a particular nasty 
one.

Limiting the ARC to, say, 16 GiB, has no effect of the high amount of 
wired memory. After a few more days, the kernel consumes virtually all 
memory, forcing processes in and out of the swap device.

stable/10 never exhibited these symptoms, even with ZFS.

I had hoped the kernel would manage its memory usage more wisely, but 
maybe it's time to set some hard limits on the kernel.

Last year, I experienced deadlocks on stable/11 systems running ZFS 
with only 1 GiB of RAM. periodic(8) and clang jobs would never be 
rescheduled, they just sat there doing nothing halfway through their 
mission and with most of their pages on the swap device. I was lucky 
enough to be able to log in and reboot the damned servers. I installed 
8 GiB of memory in each server and I never saw any deadlocks since.

Maybe we should try and help by run (virtual) machines with low 
amounts of memory and high loads to weed out these bugs, if they still 
persist.

-- 
Trond.