TCP stack lock contention with short-lived connections

[Adrian Chadd]

On 20 May 2015 at 06:27, Julien Charbon <jch at freebsd.org> wrote:
>
>  Hi,
>
> On 26/05/14 15:36, Julien Charbon wrote:
>> On 23/05/14 23:37, Navdeep Parhar wrote:
>>> On 05/23/14 13:52, Julien Charbon wrote:
>>>> On 23/05/14 14:06, Julien Charbon wrote:
>>>>> On 27/02/14 11:32, Julien Charbon wrote:
>>>>>> On 07/11/13 14:55, Julien Charbon wrote:
>>>>>>> On Mon, 04 Nov 2013 22:21:04 +0100, Julien Charbon
>>>>>>> <jcharbon at verisign.com> wrote:
>>>>>>>> I have put technical and how-to-repeat details in below
>>>>>>>> PR:
>>>>>>>>
>>>>>>>> kern/183659: TCP stack lock contention with short-lived
>>>>>>>> connections
>>>>>>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=183659
>>>>>>>>
>>>>>>>> We are currently working on this performance improvement
>>>>>>>> effort;  it will impact only the TCP locking strategy not
>>>>>>>> the TCP stack logic itself.  We will share on freebsd-net
>>>>>>>> the patches we made for reviewing and improvement
>>>>>>>> propositions;  anyway this change might also require enough
>>>>>>>> eyeballs to avoid tricky race conditions introduction in
>>>>>>>> TCP stack.
>>>>
>>>> [...]
>>
>>  Below, just for your information, more details on context of these
>> changes:
>>
>>  o The rough consensus at BSDCan was that there is a shared interest for
>> scalability improvement of TCP workloads with potential high rate of
>> connections establishment and tear-down.
>>
>>  o Our requirements for this task were:
>>   - Achieve more than 100k TCP connections per second without dropping a
>> single packet in reception
>>   - Use a strategy that does not require to change all network stacks in
>> a row (TCP, UDP, RAW, etc.)
>>   - Be able to progressively introduce better performance, leveraging
>> already in place mutex strategy
>>   - Keep the TCP stack stable (obviously)
>
>  For people interested about this short-lived TCP connection scalability
> effort, you can subscribe to the review of our latest (and biggest so
> far) change:
>
> Decompose TCP INP_INFO lock to increase short-lived connections scalability
> https://reviews.freebsd.org/D2599
>
>  The main goal of this review is ideally to start the rough discussion
> before BSDCan (and then discuss details in person at BSDCan), and ease
> tests by other people with more exotic configurations (thanks Adrian for
> your early tests).  This patch still improves the short-lived TCP
> connection rate (setup and teardown) from 60k/sec to 150k/sec.

Hi!

I'm using this in our testing lab at work. I get to around 105k/sec,
but I think that's primarily because of other lock contention in the
kernel (things doing unnecessary ioctl()s.)



-adrian