FreeBSD 8.0 - network stack crashes?

[Gavin Atkinson]

On Mon, 2009-11-02 at 10:52 -0500, Weldon S Godfrey 3 wrote:
> Up until yesterday, we have been running FreeBSD-CURRENT of 12/08.  We 
> started to see a couple months ago some very odd network behavior. 
> Something happens to the stack that causes processes accessing the network 
> to just hang.  After the problem happens, usually (but not always), you 
> can't ssh in.  Always, you can't ssh or telnet out, and nothing can access 
> the NFS shares on the server.  You can ping everything from the server. 
> You can't even do a route add, you can't ssh if you use just the IP 
> address (although pinging with hostnames it doesn't have cached or in 
> hosts table resolves).  When you try to ssh out, do a route add from the 
> box, the process just hangs.  You can't control C it at all, it hangs 
> forever.  There is nothing in dmesg or messages to indicate an issue.  I 
> try to up/down the interfaces.  In CURRENT-12/08, it may allow things to 
> work for like 30s.

Some things that would be useful:

- Does "arp -da" fix things?
- What's the output of "netstat -m" while the networking is broken?
- What does CTRL-T show for the hung SSH or route processes?
- What does "procstat -kk" on the same processes show?
- Does going to single user mode ("init 1" and killing off any leftover
processes) cause the machine to start working again?  If so, what's the
output of "netstat -m" afterwards?

If you look to be hitting some of the limits shown by "netstat -m", try
logging the date, "netstat -m" and "vmstat -m" to a file every 30
seconds or similar so that we can see if it is a memory leak, and what
may be leaking.

Gavin