ARC size constantly shrinks, then ZFS slows down extremely

[Artem Belevich]

I left my box running over the weekend (vm.kmem_size=8G) in a loop
doing a build and then deleting build results. Each loop cycle takes
about 5 hours. All in all it touches about 2GB of sources and produces
about 30GB of object files and stuff.

This morning ARC size is around 2.5G. Now and then it dips down to 1G.
I've attached graph with memory stats and ARC size.

--Artem

> ===============================================================
> Now, the same experiment, with vm.kmem_size=8G
> vm.kmem_size: 8589934592
> vfs.zfs.arc_min: 939524096
> vfs.zfs.arc_max: 7516192768
>
> ARC grows to 6.2G:
> Mem: 47M Active, 13M Inact, 7376M Wired, 31M Buf, 473M Free
>
> Then it quickly shrinks to 4.6G and grows to 6.2G again, shrinks again, etc..
>
> What's different from the previous case is that after a while ZFS
> adjusts target size (kstat.zfs.misc.arcstats.c) down to ~5.8G and
> after that ZFS size oscillates between 4.2G and 5.6G. Another
> observation -- ARC shrinking happens when system is left with ~512M of
> free memory. Yet another observation is that even with ARC peak of
> ~5.8G, system has about 7.5G wired. Where did almost 2G of difference
> go? Fragmentation?
>
> I've tried both experiments with and without L2ARC -- behavior seems
> to be the same.
>
> --Artem
>