About netstat -m: What is "mbuf+clusters out of packet secondary zone in use" ?

[Yony Yossef]

> > I'm testing an Ethernet driver on FreeBSD 6.3.
> >
> > Running netstat -m during an ethernt stress test I see that the 
> > "mbuf+clusters out of packet secondary zone in use" number 
> is growing 
> > gradualy. Problem is it never goes down after I stop the 
> test, so it's 
> > pushing the "mbufs in use" up until the following stress test 
> > iterations reach the OS limit. What does this number mean?
> 
> It seems likely that one of two things is happening:
> 
> (1) a leak of mbuf + clusters
> (2) a bug in stats on mbuf + clusters
> 
> Could you paste the output of "vmstat -z | grep -i mbuf" into 
> an e-mail? 
> That's the underlying vm stats from the zone used to generate 
> netstat -m output, and might shed more light on things.

Thanks Robert.

Here's the dump. In the beginning of the test this is the status:

Tue Dec 16 10:34:15 UTC 2008
4353/1152/5505 mbufs in use (current/cache/total)
4352/706/5058/65536 mbuf clusters in use (current/cache/total/max)
256/640 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
9792K/1700K/11492K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS
FAILURES
mbuf_packet:              256,        0,      256,      640,    12466,
0
mbuf:                     256,        0,     4097,      512,    26854,
0
mbuf_cluster:            2048,    65536,     4992,       66,     4992,
0
mbuf_jumbo_pagesize:     4096,        0,        0,        0,        0,
0
mbuf_jumbo_9k:           9216,        0,        0,        0,        0,
0
mbuf_jumbo_16k:         16384,        0,        0,        0,        0,
0


After a while the mbuf+clusters is growing. This output was taken a long
time after the test ended:

Tue Dec 16 10:38:58 UTC 2008
12873/62142/75015 mbufs in use (current/cache/total)
6780/58756/65536/65536 mbuf clusters in use (current/cache/total/max)
2337/542 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
16778K/133047K/149825K bytes allocated to network (current/cache/total)
0/77/15 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS
FAILURES
mbuf_packet:              256,        0,     2337,      542,  1939590,
15
mbuf:                     256,        0,    10536,    61600, 10221115,
0
mbuf_cluster:            2048,    65536,     7322,    58214,  3316319,
77
mbuf_jumbo_pagesize:     4096,        0,        0,        0,        0,
0
mbuf_jumbo_9k:           9216,        0,        0,        0,        0,
0
mbuf_jumbo_16k:         16384,        0,        0,        0,        0,
0