peak mbuf stat missing ... and needed ...

[Robert Watson]

On Mon, 19 Nov 2007, Juri Mianovich wrote:

> I am sorry to repost, but I cannot get any answer on this from -net or 
> -questions ... is there any answer to getting this stat ?  (see below)

Juri,

I recognize the importance of your point, and can shed a little light on why 
things are the way they are.  In FreeBSD 5, Bosko Milekic introduced MBUMA, a 
UMA-backed caching slab allocator for mbufs and related data structures 
implemented using extensions to UMA(9).  One of the properties of UMA is that 
it's possible to allocate packet storage from CPU-local caches rather than 
going to a central pool protected by central locks.  Almost all allocations 
occur this way in practice, and only intermittently return to the centra 
allocator to eithe flush many freed packets back to the central cache, or pull 
more out; this occurs when there is an imbalance in allocation and freeing 
across CPUs, such as when a pipeline occurs in packet processing over a series 
of CPUs.  As a result, there is in fact no central tracking of how many mbufs 
are currently allocated -- the central zone knows about the number currently 
not present in the zone, but that just means they are in either a per-CPU 
cache or in use, not that they are actually allocated.

The notion of peak allocation is obviously a very important one for precisely 
the reasons you identify.  The question is how best to provide it without 
seriously impacting performance *or* providing one that is potentially quite 
inaccurate.  The "current" measure is based on taking a non-atomic snapshot of 
the global allocation stats and per-CPU stats, which means potentially it can 
be very slightly inconsistent.  We don't want to update the peak stat on every 
allocation, I think, as it would be a global measure, and involve dirtying 
global cache lines and so on.  Perhaps we could be maintaining that peak value 
whenever CPUs go back to the global pool from a per-CPU cache, since the right 
locks will be held anyway...  I don't see this being fixed for 6.3 or 7.0 
given their proximity, but I will investigate a fix for later releases.  Could 
you file a feature request PR on this, and forward me the PR receipt so I can 
take ownership of it?

Thanks,

Robert N M Watson
Computer Laboratory
University of Cambridge

>
> -----
>
> FreeBSD 4.x, netstat -m:
>
> 70/4336/26624 mbufs in use (current/peak/max)
>
> Never any doubt - if peak=max, I hit the limit.  Super
> useful.  Furthermore, by watching the peak I can see
> when I am getting close, rather than waiting for
> denied requests to pile up after the fact.
>
> FreeBSD 6.x, netstat -m:
>
> 524/826/1350 mbufs in use (current/cache/total)
>
> So ... how do I see peak mbufs in FreeBSD 6.x ?
>
> Thanks.
>
>
>
>      ____________________________________________________________________________________
> Get easy, one-click access to your favorites.
> Make Yahoo! your homepage.
> http://www.yahoo.com/r/hs
>
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"
>