maximum number of outgoing connections

[Igor Sysoev]

On Mon, Aug 20, 2007 at 08:34:43PM +0400, Igor Sysoev wrote:

> On Mon, Aug 20, 2007 at 05:19:14PM +0100, Tom Judge wrote:
> 
> > Igor Sysoev wrote:
> > >It seems that FreeBSD can not make more than
> > >
> > >net.inet.ip.portrange.last - net.inet.ip.portrange.first
> > >
> > >simultaneous outgoing connections, i.e., no more than about 64k.
> > >
> > >If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then
> > >connect() to an external address returns EADDRNOTAVAIL.
> > >
> > >net.inet.ip.portrange.randomized is 0.
> > >
> > >sockets, etc. are enough:
> > >
> > >ITEM        SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
> > >socket:      356,   204809,    13915,   146443, 148189452,        0
> > >inpcb:       180,   204820,    20375,   137277, 147631805,        0
> > >tcpcb:       464,   204800,    13882,   142102, 147631805,        0
> > >tcptw:        48,    41028,     6493,    11213, 29804665,        0
> > >
> > >I saw it on 6.2-STABLE.
> > >
> > >
> > 
> > In an ideal world (Not sure if this is quite correct for FreeBSD) TCP 
> > connections are tracked with a pair of tupels  source-addr:src-port -> 
> > dst-addr:dst-port
> > 
> > As your always connecting to the same destination service 127.0.0.1:80 
> > and always from the same source IP 127.0.0.1 then you only have one 
> > variable left to change, the source port.  If you where to use the hole 
> > of the whole of the port range minus the reserved ports you would only 
> > ever be able to make 64512 simultaneous connections.  In order to make 
> > more connections the first thing that you may want to start changing is 
> > the source IP. If you added a second IP to you lo0 interface (say 
> > 127.0.0.2) and used a round robin approach to making your out bound 
> > connections then you could make around 129k outbound connections.
> 
> Connections to 127.0.0.1 were via lo0, external connections are via bge0.
> 
> > I am not sure if there are any other constraints that need to be taken 
> > into account such as the maximum number of sockets, RAM etc....
> 
> No, there are no constraints in memory, sockets, mbufs, clusters, etc.
> If there's contraint in memory, then FreeBSD simply panics.
> If there's contraint in mbuf clusters, then process stucks in zonelimit
> state forever.
> 
> I suspect that local address in in_pcbbind_setup() is 0.0.0.0 so there
> is 64K limit.

Recently I looked the issue again and find (with Ruslan Ermilov's help)
that if I set the SO_REUSEADDR option, FreeBSD allows to use the same
local port for different connections:

        192.168.1.1:5000  > 10.0.0.1:80
        192.168.1.1:5000  > 10.0.0.2:80

Linux allows this by default. I believe FreeBSD should set SO_REUSEADDR
internally while connect().

BTW, bind()ing socket to a local address does not help.


-- 
Igor Sysoev
http://sysoev.ru/en/