mbuma: network buffers & UMA

[Bosko Milekic]

The post below has received exactly zero attention on the
freebsd-current mailing list.  I'm assuming this is because there are
other exciting threads going on there right now, such as style debates
and lovely feature requests. :-)

I would like to commit this RSN.  I need feedback from the Release
Engineering people on how close we are to a RELEASE and if it's OK for
me to break netstat -m stats for a few weeks or, if not, what the
appropriate course of action is.


Fellow earthlings,

    I would like to request some testing on diverse architectures of the
    following patch (~130K):

    http://people.freebsd.org/~bmilekic/code/mbuma-v2.diff

    In short, the above accomplishes several things and is the result of
    one of my now mature p4 repositories which I have been able to again
    address following a generous hardware donation.

    This basically gets rid of the existing mbuf allocator and replaces
    it with some routines which get hooked on top of UMA, the existing
    general-purpose SMP-friendly allocator in -CURRENT, after adding
    some extensions to UMA which hopefully make allocations faster than
    they would be without them.

    The current list of implications (whether good or bad is for you to
    decide):

    - NMBCLUSTERS, as a compile-time option, is now gone.  The
      kern.ipc.nmbclusters tunable and sysctl still exist and are both
      read+write.  Currently, the sysctl does not effect anything but
      will soon be made to allow for runtime (dynamic) modification of
      the maximum number of mbuf clusters allocatable cap.  If you set
      the tunable to zero, though, the number of clusters allocatable is
      unbounded and the system will scale according to demand.  Be
      careful... leaving these unbounded may not be what you really
      want.

    - Memory allocated to network buffers can (and now will) be
      reclaimed by UMA when required and, for example, after a large
      network spike.

    - Current on-par performance with the existing allocator.  Soon some
      modifications to UMA that improve performance drastically in the
      common case may change that, though - for the better.

    - For developers: M_WAIT behavior now is the same as for all other
      UMA allocations, and is no longer special for network buffers.

    - Currently netstat(1) mbuf stats ('netstat -m') are broken and
      instead the following is displayed:
 
       fermat% netstat -m
       Mbuf statistics are currently unavailable via netstat(1), see
       UPDATING.
       This is only a TEMPORARY measure in -CURRENT.
       For current mbuf and cluster allocation stats, see sysctl vm.zone
  
      I don't plan to fix them prior to doing some other work (which may
      change the way stats work, anyway) in UMA.  Hopefully this won't
      be too much of a problem for most of you.

    - So far has been very stable on my dual Athlon and a friend's x86
      UP machine (I think it's UP, anyway).  Some other architectures,
      particular 64-bit ones, would be worth testing.

    - Some things (e.g., dev/nsp) I just noticed might be broken by the
      above changes due to slight time-delay between the vendor p4 branch
      and the CVS checked out repo I used to generate the final diff.
      If something you're using is broken by the diff, it is only broken
      slightly, and you'll notice it immediately during kernel compile,
      so you can fix it (it's generally just changing some includes) or
      Email it to me and I'll send you a custom diff.  This only
      applies to the current diff posted above and will not be
      committed.


  That's all I could think of right now.  I would like to commit this
  soon as I have other things in the pipeline that build on top of it.
  I should also mention that there are various other implications in
  addition to the above but are more relevant to the developer than the
  user, so they'll probably appear in a really large commit log.
  
  Reviews and constructive criticism is both welcomed and appreciated.
  Please leave me in the To: line - although I am subscribed to
  -current, I'd like to see feedback in my main box.

Regards,
--
Bosko Milekic  *  bmilekic at technokratis.com  *  bmilekic at FreeBSD.org
TECHNOkRATIS Consulting Services  *  http://www.technokratis.com/

"It is impossible for anyone to begin to learn what he believes
 he already knows."  -- Epictetus (c.a.d. 55-c135)