nmbclusters: how do we want to fix this for 8.3 ?

[Luigi Rizzo]

On Wed, Feb 22, 2012 at 11:56:29AM -0800, Jack Vogel wrote:
> Using igb and/or ixgbe on a reasonably powered server requires 1K mbuf
> clusters per MSIX vector,
> that's how many are in a ring. Either driver will configure 8 queues on a
> system with that many or more
> cores, so 8K clusters per port...
> 
> My test engineer has a system with 2 igb ports, and 2 10G ixgbe, this is
> hardly heavy duty, and yet this
> exceeds the default mbuf pool on the installed kernel (1024 + maxusers *
> 64).
> 
> Now, this can be immediately fixed by a sysadmin after that first boot, but
> it does result in the second
> driver that gets started to complain about inadequate buffers.
> 
> I think the default calculation is dated and should be changed, but am not
> sure the best way, so are
> there suggestions/opinions about this, and might we get it fixed before 8.3
> is baked?

I have hit this problem recently, too.
Maybe the issue mostly/only exists on 32-bit systems.
Here is a possible approach:

1. nmbclusters consume the kernel virtual address space so there 
   must be some upper limit, say 

        VM_LIMIT = 256000 (translates to 512MB of address space)

2. also you don't want the clusters to take up too much of the available
   memory. This one would only trigger for minimal-memory systems,
   or virtual machines, but still...

        MEM_LIMIT = (physical_ram / 2) / 2048

3. one may try to set a suitably large, desirable number of buffers

        TARGET_CLUSTERS = 128000

4. and finally we could use the current default as the absolute minimum

        MIN_CLUSTERS = 1024 + maxusers*64

Then at boot the system could say

        nmbclusters = min(TARGET_CLUSTERS, VM_LIMIT, MEM_LIMIT)

        nmbclusters = max(nmbclusters, MIN_CLUSTERS)


In turn, i believe interfaces should do their part and by default
never try to allocate more than a fraction of the total number
of buffers, if necessary reducing the number of active queues.

what do people think ?

	cheers
	luigi