Socket leak (Was: Re: What triggers "No Buffer Space) Available"?

[Matthew Dillon]

:*groan*  why couldn't this be happening on a server that I have better remote 
:access to? :(
:
:But, based on your explanation(s) above ... if I kill off all of the jail(s) on 
:the machine, so that there are minimal processes running, shouldn't I see a 
:significant drop in the number of sockets in use as well?  or is there 
:something special about single user mode vs just killing off all 'extra 
:processes'?
:
:- ----
:Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)

    Yes, you can.  Nothing special about single user... just kill all
    the processes that might be using sockets.  Killing the jails is a good
    start.

    If you are running a lot of jails then I would strongly suspect that
    there is an issue with file desciptor passing over unix domain sockets.
    In particular, web servers, databases, and java or other applets could
    be the culprit.

    Other possibilities... you could just be running out of file descriptors
    in the file descriptor table.

    use vmstat -m and vmstat -z too... find out what allocates the socket
    memory and see what it reports.  Check your mbuf allocation statistics
    too (netstat -m).  Damn, I wish that information were collected
    on a per-jail basis but I don't think it is.  Look at all the memory
    statistics and check to see if anything is growing unbounded over a
    long period of time (verses just growing into a cache balance).  Create
    a cron job that dumps memory statistics once a minute to a file then
    break each report with a clear-screen sequence and cat it in a really
    big xterm window.

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>