mbuf revision, testers/comments wanted.

[Jeff Roberson]

http://people.freebsd.org/~jeff/mbuf_ref2.diff

Hello,

I have been experimenting with different revisions to the mbuf api to 
improve performance and simplify code.  This patch is the first of several 
proposed steps towards those goals.  The aim of this patch is two fold;

1)  Revising the reference counting system so that we can eliminate 
reference uma zones and the significant uma_find_refcnt() costs in some 
workloads.  This is done by making all mbufs reference counted and using 
the owning mbuf's ref for the ext_ref.  In this model we never reference 
data, we only reference other mbufs owning the data.

2)  Improve allocation and free performance by reducing the special cases 
in the format and using inlines when appropriate.  In particular, the 
simplification of the m_ext structure yields less code and confusion for 
dealing with external storage on free.  The ctor/dtor mbuf routines are no 
longer used.  A zone pointer and length was added to struct mbuf to 
simplify free and size calculations.

A number of routines were made much, much simpler by the addition of a 
16bit size field.  Previously we dependend on calculating the size by 
figuring out if it was an ext, pkthdr, or standard mbuf.  Ultimately, this 
patch moves us closer to having a size agnostic mbuf which we can use to 
experiment with different allocation sizes or even backending to malloc 
for dynamically sized mbufs.

I would appreciate testing feedback from varied workloads to make sure 
there are no bugs before I go forward with this.  I have tested only host 
oriented networking with a few drivers.  It is not anticipated that there 
will be any significant incompatibilities introduced with this round but 
there is always that possibility.

Thanks,
Jeff