zonelimit issues...

[Igor Sysoev]

On Mon, Apr 21, 2008 at 05:16:28PM +0900, gnn at freebsd.org wrote:

> At Mon, 21 Apr 2008 16:46:00 +0900,
> gnn at freebsd.org wrote:
> > 
> > At Sun, 20 Apr 2008 10:32:25 +0100 (BST),
> > rwatson wrote:
> > > 
> > > 
> > > On Fri, 18 Apr 2008, gnn at freebsd.org wrote:
> > > 
> > > > I am wondering why this patch was never committed?
> > > >
> > > > http://people.freebsd.org/~delphij/misc/patch-zonelimit-workaround
> > > >
> > > > It does seem to address an issue I'm seeing where processes get into the 
> > > > zonelimit state through the use of mbufs (a high speed UDP packet receiver) 
> > > > but even after network pressure is reduced/removed the process never gets 
> > > > out of that state again.  Applying the patch fixed the issue, but I'd like 
> > > > to have some discussion as to the general merits of the approach.
> > > >
> > > > Unfortunately the test that currently causes this is tied very tightly to 
> > > > code at work that I can't share, but I will hopefully be improving mctest to 
> > > > try to exhibit this behavior.
> > > 
> > > When you take all load off the system, do mbufs and clusters get properly 
> > > freed back to UMA (as visible in netstat -m)?  If not, continuing to bump up 
> > > against the zonelimit would suggest an mbuf/cluster leak, in which case we 
> > > need to track that bug.
> > > 
> > 
> > This is unclear as the process that creates the issue opens 50 UDP
> > multicast sockets with very large socket buffers.  I am investigating
> > this aspect some more.
> > 
> 
> OK, yes, the clusters etc. go back to normal when the incoming
> pressure is released.  I do not believe we have a cluster/mbuf leak.

There is no cluster/mbuf leak.

The problem that FreeBSD has small KVA space: only 2G even on amd64 32G
machines.

So with

vm.kmem_size=1G
# 64M KVA
kern.maxbcache=64M
# 4M KVA
kern.ipc.maxpipekva=4M


I can use something like this:

# 256M KVA/KVM
kern.ipc.nmbjumbop=64000
# 216M KVA/KVM
kern.ipc.nmbclusters=98304
# 162M KVA/KVM
kern.ipc.maxsockets=163840
# 8M KVA/KVM
net.inet.tcp.maxtcptw=163840
# 24M KVA/KVM
kern.maxfiles=204800


-- 
Igor Sysoev
http://sysoev.ru/en/