zfs + uma

[Robert Watson]

On Fri, 17 Sep 2010, Andre Oppermann wrote:

>> Although keeping free items around improves performance, it does consume 
>> memory too.  And the fact that that memory is not freed on lowmem condition 
>> makes the situation worse.
>
> Interesting.  We may run into related issues with excessive mbuf (cluster) 
> caching in the per-cpu buckets as well.
>
> Having a general solutions for that is appreciated.  Maybe the size of the 
> free per-cpu buckets should be specified when setting up the UMA zone.  Of 
> certain frequently re-used elements we may want to cache more, other less.

I've been keeping a vague eye out for this over the last few years, and 
haven't spotted many problems in production machines I've inspected.  You can 
use the umastat tool in the tools tree to look at the distribution of memory 
over buckets (etc) in UMA manually.  It would be nice if it had some automated 
statistics on fragmentation however.  Short-lived fragmentation is likely, and 
isn't an issue, so what you want is a tool that monitors over time and reports 
on longer-lived fragmentation.

The main fragmentation issue we've had in the past has been due to 
mbuf+cluster caching, which prevented mbufs from being freed usefully in some 
cases.  Jeff's ongoing work on variable-sized mbufs would entirely eliminate 
that problem...

Robert