kern/183659: [tcp] TCP stack lock contention with short-lived connections

[Charbon, Julien]

The following reply was made to PR kern/183659; it has been noted by GNATS.

From: "Charbon, Julien" <jcharbon at verisign.com>
To: bug-followup at FreeBSD.org
Cc:  
Subject: Re: kern/183659: [tcp] TCP stack lock contention with short-lived
 connections
Date: Mon, 17 Mar 2014 15:54:03 +0100

   Just a follow-up that updates lock profiling results with
 short-lived TCP connection traffic on FreeBSD-10.0 RELEASE:
 (Previous results were made on FreeBSD-9.2 RELEASE)
 
 o FreeBSD-10 RELEASE:
 
 # sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -5
 debug.lock.prof.stats:
       max  wait_max       total  wait_total       count    avg wait_avg cnt_hold cnt_lock name
        37    321900     3049892    13033648      610019      4     21  0 588013 sys/netinet/tcp_input.c:778 (rw:tcp)     tcp_input() (SYN|FIN|RST)
        51    115462     3240265    12270984      553157      5     22  0 545293 sys/netinet/tcp_input.c:1013 (rw:tcp)    tcp_input() (state != ESTABLISHED)
        29     62577     1170617     8754815      305885      3     28  0 296845 sys/netinet/tcp_usrreq.c:728 (rw:tcp)    tcp_usr_close()
         6     62645      146544     8548857      292058      0     29  0 283587 sys/netinet/tcp_usrreq.c:984 (rw:tcp)    tcp_usr_shutdown()
        11     62595      198811     6525067      309009      0     21  0 304522 sys/netinet/tcp_usrreq.c:635 (rw:tcp)    tcp_usr_accept()
 
   - If lock contention spots moved a little between 9.2 and 10.0, nothing
 major as the top 5 still belongs to (rw:tcp) lock (a.k.a. TCP INP_INFO).
 
 o FreeBSD-10 RELEASE + PCBGROUP kernel option (by popular demand):
 
 # sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -5
 debug.lock.prof.stats:
       max  wait_max       total  wait_total       count    avg wait_avg cnt_hold cnt_lock name
        58     84250     2970633    13154832      622401      4     21  0 598964 sys/netinet/tcp_input.c:778 (rw:tcp)     tcp_input() (SYN|FIN|RST)
        47    224326     3375328    12945466      562451      6     23  0 554567 sys/netinet/tcp_input.c:1013 (rw:tcp)    tcp_input() (state != ESTABLISHED)
        22     84332     1193078     9693951      311555      3     31  0 302420 sys/netinet/tcp_usrreq.c:728 (rw:tcp)    tcp_usr_close()
         6     84307      151411     9137383      298120      0     30  0 289496 sys/netinet/tcp_usrreq.c:984 (rw:tcp)    tcp_usr_shutdown()
        15     84351      201705     6504520      314353      0     20  0 310270 sys/netinet/tcp_usrreq.c:635 (rw:tcp)    tcp_usr_accept()
 
   - No changes at all in first ranks by using PCBGROUP option on
 FreeBSD-10 RELEASE. I have indeed checked that PCBGROUP was in
 use as at #36 rank there is the specific pcbgroup lock:
 
        11         9      289817        4815     1505626      0      0  0  16054 sys/netinet/in_pcb.c:1530 (sleep mutex:pcbgroup)
 
 o FreeBSD-10 RELEASE + current lock mitigation patches [1][2]:
 
 # sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -20
 debug.lock.prof.stats:
       max  wait_max       total  wait_total       count    avg wait_avg cnt_hold cnt_lock name
        29       297     3781629    13476466      734686      5     18  0 715214 sys/netinet/tcp_input.c:778 (rw:tcp)     tcp_input() (SYN|FIN|RST)
        35       287     3817278    12301410      672907      5     18  0 669324 sys/netinet/tcp_input.c:1013 (rw:tcp)    tcp_input() (state != ESTABLISHED)
        18       170     1392058     2494823      367131      3      6  0 357888 sys/netinet/tcp_usrreq.c:719 (rw:tcp)    tcp_usr_shutdown()
         7       141      182209     2433120      350488      0      6  0 344878 sys/netinet/tcp_usrreq.c:975 (rw:tcp)    tcp_usr_close()
        10       259       26786      933073       38101      0     24  0  37624 sys/netinet/tcp_timer.c:493 (rw:tcp)     tcp_timer_rexmt()
 
   - No more tcp_usr_accept() (expected)
 
   o Global results:  Maximum short-lived TCP connection rate without dropping a single packet:
 
   - FreeBSD 10.0 RELEASE:             40.0k
   - FreeBSD 10.0 RELEASE + PCBGROUP:  40.0k
   - FreeBSD 10.0 RELEASE + patches:   56.8k
 
 [1] Decrease lock contention within the TCP accept case by removing
   the INP_INFO lock from tcp_usr_accept.
 http://svnweb.freebsd.org/base?view=revision&revision=261242
 
 [2] tw-clock-v2.patch attached in:
 http://lists.freebsd.org/pipermail/freebsd-net/2014-March/038124.html
 
 --
 Julien