sysctl kern.ipc.somaxconn limit 65535 why?

[Dan The Man]

On Wed, 4 Jan 2012, Dan The Man wrote:

>
> On Wed, 4 Jan 2012, Chuck Swiger wrote:
>
>> On Jan 4, 2012, at 1:49 PM, Arnaud Lacombe wrote:
>>> Hi,
>>> 
>>> On Wed, Jan 4, 2012 at 4:42 PM, Chuck Swiger <cswiger at mac.com> wrote:
>>>> On Jan 4, 2012, at 1:03 PM, Dan The Man wrote:
>>>>>> However, I'm not convinced that it is useful to do this.  At some 
>>>>>> point, you are better off timing out and retrying via exponential 
>>>>>> backoff than you are queuing hundreds of thousands of connections in 
>>>>>> the hopes that they will eventually be serviced by something sometime 
>>>>>> considerably later.
>>>>> 
>>>>> I agree completely, in practical application this makes sense, but why 
>>>>> should the OS dictate not being able to temporarily set that setting 
>>>>> higher in order to fully benchmark the application at 100k+ in the 
>>>>> listen queue if the developer so chooses? I think that alone should be a 
>>>>> good reason, to make freebsd developer friendly.
>>>> 
>>>> The job of the OS is to manage resources on behalf of the users and 
>>>> processes using the system.
>>> 
>>> No. The job of the OS is to service the user with the resource
>>> available, not constrict the user within some arbitrary predefined
>>> wall when there is still plenty of room available. If resource become
>>> scarce, then take action.
>> 
>> It is not arbitrary.  Systems ought to provide sensible limits, which can 
>> be adjusted if needed and appropriate.  The fact that a system might have 
>> 50,000 file descriptors globally available does not mean that it would be 
>> OK for any random process to consume half of them, even if there is still 
>> adequate room left for other tasks.  It's common for "ulimit -n" to be set 
>> to 256 or 1024.
>
> Sensibly limits means a sensible stock default, not imposing an OS limit on 
> what admin/developer can set on his own hardware.
>
> With the new IBM developments underway of 16 core atom processors and 
> hundreds of gigabytes of memory, surely a backlog of 100k is manageable. Or 
> what about the future of 500 core systems with a terrabyte of memory, 100k 
> listen queue could be processed instantly.
>
>> 
>>>> Some developers feel that VM means that the system should always claim 
>>>> have more memory available, but always saying "yes" isn't "managing 
>>>> resources".  I'd rather have the OS return a null pointer and set ENOMEM 
>>>> when someone tries to malloc() more memory than the system (including 
>>>> swap, VM overcommit, etc) has, and I expect developers to code well 
>>>> enough to handle malloc() failures.
>>> 
>>> this is merely a policy issue, not yours to impose.
>> 
>> If we're speaking of machines which I administer, it is a policy issue that 
>> would be mine to impose.
>> If we're speaking of someone else's machines, then they can set their own 
>> policies as they please.
>> 
>>>> Setting the listen queue to an arbitrarily high value isn't useful, and 
>>>> developers would be better advised to pay attention to best practices in 
>>>> the face of a massive connection backlog.
>>> 
>>> Stress-testing isn't about "best practice". It is about shaking enough
>>> the system to highlight weak point.
>> 
>> Yes.  If the system doesn't handle connectivity problems via something like 
>> exponential backoff, then the weak point is poor software design and not 
>> FreeBSD being unwilling to set the socket listen queue to a value in the 
>> hundreds of thousands.
>> 
>
> I think what me and Arnaud are trying to say here, is let freebsd use a 
> sensible default value, but let the admin dictate the actual policy if he so 
> chooses to change it for stress testing, future proofing or anything else.
>

How about a sensible solution? I think everyone has been making valid 
points here, about sensible limits for all programs on system and per 
application limit changes.

How about changing the hard limit high, and an application can make the 
soft limit higher as it sees fit, its a win win, like ulimit does with 
openfiles and such.



Dan.

--
Dan The Man
CTO/ Senior System Administrator
Websites, Domains and Everything else
http://www.SunSaturn.com
Email: Dan at SunSaturn.com