IPsec performace - netisr hits %100

[Özkan KIRIK]

Hello,

I'm using FreeBSD stable/12 built world on 12 April 2021.
my setup is:
[freebsd host cc0] <--------> [cc1 - same freebsd, but jail]

without IPsec, I can achieve easily to 20Gbps. (test was run with different
source IPs using multiple iperf to scale across multiple queues)
My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio
T62100-LP-CR.

But with IPsec, throughput is limited to 2Gbps (with ccr) and only one
netisr thread hits %100 cpu.
with aesni throughput is 1,4 Gbps
with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x)
with CCR throughput is 2,0 Gbps (t6nex0)
But always bottleneck is netisr.

Is there any way to workaround this netisr bottleneck ?
I tried to switch net.isr.dispatch to deferred and hybrid, but performance
drops a bit.

my configuration is below:

net.isr.numthreads: 4
net.isr.maxprot: 16
net.isr.defaultqlimit: 8192
net.isr.maxqlimit: 1000000
net.isr.bindthreads: 1
net.isr.maxthreads: 4
net.isr.dispatch: direct

net.inet.ipsec.async_crypto: 1

FreeBSD Host:
kldload ccr # Chelsio Crypto Accelerator
ifconfig lo1 create 172.16.68.1/24 up
ifconfig cc0 192.168.1.3/24 up
ifconfig ipsec0 create reqid 100
ifconfig ipsec0 inet tunnel 192.168.1.3 192.168.1.5
ifconfig ipsec0 inet 172.16.0.3/16 172.16.0.5
setkey -c << EOF
add 192.168.1.3 192.168.1.5 esp 10000 -m tunnel -u 100 -E aes-gcm-16
"VerySecureKey!!10000";
add 192.168.1.5 192.168.1.3 esp 10001 -m tunnel -u 100 -E aes-gcm-16
"VerySecureKey!!20000";
EOF
route add 172.16.70.0/24 172.16.0.5
iperf -s

FreeBSD Jail:
jail -c name=client persist vnet vnet.interface=cc1 host.hostname=client
jexec client sysctl net.inet.ipsec.async_crypto=1
jexec client ifconfig lo1 create 172.16.70.1/24 up
jexec client bash -c 'for i in $(seq 2 10); do ifconfig lo1 172.16.70.$i/32
alias; done'
jexec client ifconfig cc1 192.168.1.5/24 up
jexec client ifconfig ipsec0 create reqid 200
jexec client ifconfig ipsec0 inet tunnel 192.168.1.5 192.168.1.3
jexec client ifconfig ipsec0 inet 172.16.0.5/16 172.16.0.3
jexec client setkey -c
add 192.168.1.3 192.168.1.5 esp 10000 -m tunnel -u 200 -E aes-gcm-16
"VerySecureKey!!10000";
add 192.168.1.5 192.168.1.3 esp 10001 -m tunnel -u 200 -E aes-gcm-16
"VerySecureKey!!20000";
^D
jexec client route add 172.16.68.0/24 172.16.0.3

jexec client bash -c 'for i in $(seq 1 10); do (iperf -B 172.16.70.$i -c
172.16.68.1 -P 2 | grep SUM &); done'
---------------------------------

top -azSHj
  PID   JID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU
COMMAND
   11     0 root        -72    -     0B  1120K CPU2     2   0:26 100.00%
[intr{swi1: netisr 2}]
   11     0 root        -92    -     0B  1120K CPU4     4   0:36  72.55%
[intr{irq295: t6nex0:0a0}]
    3     0 root        -16    -     0B    16K CPU8     8   0:13  51.11%
[crypto returns 0]
   11     0 root        -92    -     0B  1120K WAIT    11   0:18  43.40%
[intr{irq297: t6nex0:0a2}]
   14     0 root        -16    -     0B    16K crypto  14   0:09  33.43%
[crypto returns 8]
   11     0 root        -92    -     0B  1120K WAIT    12   0:11  21.17%
[intr{irq307: t6nex0:1a2}]
 9049     1 root         32    0    23M  4356K CPU14   14   0:00   7.50%
iperf -B 172.16.70.9 -c 172.16.68.1 -P 2{iperf}
 9040     1 root         30    0    23M  4356K sbwait   8   0:00   6.92%
iperf -B 172.16.70.6 -c 172.16.68.1 -P 2{iperf}
 9043     1 root         33    0    23M  4356K sbwait   8   0:00   6.90%
iperf -B 172.16.70.7 -c 172.16.68.1 -P 2{iperf}
 9046     1 root         32    0    23M  4356K sbwait  11   0:00   6.72%
iperf -B 172.16.70.8 -c 172.16.68.1 -P 2{iperf}
 9031     1 root         24    0    23M  4356K sbwait   6   0:00   6.40%
iperf -B 172.16.70.3 -c 172.16.68.1 -P 2{iperf}
 9037     1 root         29    0    23M  4356K sbwait  11   0:00   6.05%
iperf -B 172.16.70.5 -c 172.16.68.1 -P 2{iperf}
 9037     1 root         29    0    23M  4356K sbwait   6   0:00   5.82%
iperf -B 172.16.70.5 -c 172.16.68.1 -P 2{iperf}
 9025     1 root         22    0    23M  4356K sbwait  14   0:00   5.71%
iperf -B 172.16.70.1 -c 172.16.68.1 -P 2{iperf}
 9052     1 root         32    0    23M  4356K sbwait  10   0:00   5.24%
iperf -B 172.16.70.10 -c 172.16.68.1 -P 2{iperf}

If you need more information, i can provide.

Regards,
Özkan