BETA3: network processes hang in keglimit (unkillable)

[Pyun YongHyeon]

On Fri, Sep 11, 2009 at 07:24:10PM +0200, Pieter de Goeje wrote:
> I just experienced a complete loss of network, where every network related 
> process would sleep on "keglimit". I was unable to recover from this 
> situation. I tried killing the processes with SIGKILL, but it didn't work, 
> and I tried to bring the network interface (em) down and up again, which also 
> didn't work. A reboot "solved" the problem. The machine was handling some 
> major traffic, both TCP and UDP.
> 
> Apparently the system was waiting on some resource to free up, but what could 
> it be? How can this situation be avoided in the future?
> 
> uname: 8.0-BETA3 FreeBSD 8.0-BETA3 #1: Sun Aug 30 01:23:36 CEST 2009 amd64
> 

Both em(4) and igb(4) had mbuf leak bug which was recently fixed by
Jack. Check mbuf statistics with "netstat -m" and see whether
you've reached mbuf resource limit(see 4K jumbo cluster counter).
The leak can be easily seen on UDP traffic, especially NFS over UDP.