TCP stack lock contention with short-lived connections

[Julien Charbon]

  Hi list,

  just a follow-up of vBSDCon discussions about FreeBSD TCP performances  
with short-lived connections.  In summary:

  Using a single NIC receive queue, the TCP stack performance results with  
short-lived connections on FreeBSD are great:  We got 52,000 (52k) TCP  
queries per second (QPS) before being CPU bound, we achieved the same  
result on Linux.  However, when configuring 4 NIC receive queues each  
bound to a different core of the same CPU, results are lower than  
expected:  We got 56k QPS, where we reached 200k QPS on Linux on same  
hardware.  Basically, FreeBSD TCP stack does not scale well across  
multiple receive queues/CPU cores due to a lock contention.  I have put  
technical and how-to-repeat details in below PR:

kern/183659: TCP stack lock contention with short-lived connections
http://www.freebsd.org/cgi/query-pr.cgi?pr=183659

  We are currently working on this performance improvement effort;  it will  
impact only the TCP locking strategy not the TCP stack logic itself.  We  
will share on freebsd-net the patches we made for reviewing and  
improvement propositions;  anyway this change might also require enough  
eyeballs to avoid tricky race conditions introduction in TCP stack.

  Thanks.

--
Julien