Inactive vs ZFS ARC memory priority

From: Marko Turk <markoml_at_markoturk.info>
Date: Fri, 25 Mar 2022 13:06:55 UTC
Hi,

I have a workstation with 32GB or RAM, running recent 12.3-STABLE.
vfs.zfs.arc_max is set to "16G".

After a few hours of uptime I get this memory usage:

Mem: 327M Active, 3007M Inact, 58M Laundry, 20G Wired, 7986M Free
ARC: 15G Total, 1723M MFU, 12G MRU, 9032K Anon, 246M Header, 987M Other
     13G Compressed, 31G Uncompressed, 2.44:1 Ratio
Swap: 4096M Total, 4096M Free

All looks good, ZFS ARC is used almost completely up to arc_max. And it stays
like this for days.

Then I do a make buildworld & buildkernel. And I get this:

Mem: 663M Active, 20G Inact, 879M Laundry, 8560M Wired, 1445M Free
ARC: 4351M Total, 984M MFU, 2176M MRU, 14M Anon, 37M Header, 1140M Other
     2453M Compressed, 5405M Uncompressed, 2.20:1 Ratio
Swap: 4096M Total, 4096M Free

A lot of memory is in Inactive, and ARC decreased. I think this is still OK.

But what I expect to happen eventually is that ARC will grow again and push old
data from Inactive. This never happens. I can leave it for days with a lot of
disk activity and nothing changes.

Seems like old data in Inactive has higher priority than ARC. Why is that so?
Can I improve this somehow?

If I do something memory intesive, Inacive will decrease. This works as
expected, I get something like this:

Mem: 659M Active, 16G Inact, 851M Laundry, 9337M Wired, 4343M Free
ARC: 5144M Total, 1207M MFU, 2712M MRU, 7275K Anon, 82M Header, 1135M Other
     2885M Compressed, 6647M Uncompressed, 2.30:1 Ratio
Swap: 4096M Total, 4096M Free

After that the ARC will grow and fill the empty memory. But on it's own it will
never push Inactive down.

Why is old data in Inactive higher priority than new data in ARC?

Regards,
Marko