mbuf clusters exhaustion & keglimit

[Eugene Grosbein]

Hi!

I'm running several loaded PPPoE access servers based on FreeBSD 8.2-STABLE/amd64
with em and igb network interfaces and 4GB RAM. No memory-intensive tasks
other than routing about 2Gbit/s (1G "in" and a bit less "out").

kern.ipc.nmbclusters is set to 100000 in /etc/sysctl.conf and several months
I had no problems with mbufs. Last week one of the routes stopped serviceing
users for several hours but responded to pings and console was alive.
Outgoing ping worked fine too but any process trying to use
TCP or UDP kernel service got stuck in "keglimit" state.

I've dropped to KDB from console, ran "call doadump", got full crashdump,
returned from KDB, saved crashdump and tried to reboot cleanly.

mpd5 failed to stop within 30 seconds timeout but file systems
were unmounted cleanly and system rebooted.

"vmstat -z -M vmcore" says that system was out of mbuf clusters:

ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
mbuf_cluster:            2048,   100000,   100000,        0, 18897242,   317691

After that I've created graphs of mbuf cluster usage for all my routers
and see no apparent leaks.

The question is: how much kernel memory is it safe to dedicate to mbuf clusters?
This system still runs with 100000 mbuf clusters maximum:

Mem: 65M Active, 2759M Inact, 455M Wired, 31M Cache, 398M Buf, 435M Free

It seems, 100000 mbuf clusters take only 207MB (2048+256 bytes for each), do they?

Eugene Grosbein