Big physically contiguous mbuf clusters

[Garrett Wollman]

In article <20140129231138$3db6 at grapevine.csail.mit.edu>,
nparhar at gmail.com writes:

>I think this would be very useful.  For example, a zone_jumbo32 would
>hit a sweet spot -- enough to fit 3 jumbo frames and some loose change
>for metadata.  I'd like to see us improve our allocators and VM system
>to work better with larger contiguous allocations, rather than
>deprecating the larger zones.  It seems backwards to push towards
>smaller allocation units when installed physical memory in a typical
>system continues to rise.

In order to resist fragmentation, you need to be willing to dedicate
some partition of physical memory to larger allocations.  That's fine
for a special-purpose device like a switch, but is not so good for a
general-purpose operating system.  But if you were willing to reserve,
say, 1/64th of physical memory at boot time, make it all direct-mapped
using superpages, and allocate it in fixed-power-of-two-sized chunks,
you would probably get a performance win.  But the chunks *have* to be
fixed-size, otherwise you are nearly guaranteed to get your arena
checkerboarded.  I'd consider giving 2 GB on a 128-GB machine for
that.

For NFS performance, you'd probably want to be able to take a whole
chunk, read the desired data into it in a single VOP, then pass the
whole thing to the socket layer wrapped in an mbuf.

-GAWollman