Is ARC too conservative?

[Paul]

Hello team,

On numerous occasions we have observed a strange thing. Basically, we observer it after every restart of long running server.

The thing is that disk busy % (both SSD and HDD) drops dramatically (given the identical overall server load): by 40% - 50%. 
Then slowly, over period of few weeks, busy percentage grows to the previous value where it stabilises.

You don't even have to restart server to reproduce this thing. It is enough to drop ARC size to it's minimum value by 
applying synthetic memory pressure.


Could it be that ARC is too conservative regarding the data that is already in cache so that it gives less opportunity for
new data to be added? I strongly suspect it's the MFU that is too conservative. How long may the blocks that have a lot 
of hits stay there? How the score is calculated and is there a way to tweak the formula via sysctl?

Thanks.

Best regards,

Paul