IPsec performace - netisr hits %100

[Mark Johnston]

On Sat, May 01, 2021 at 12:31:57AM +0300, Özkan KIRIK wrote:
> Hello again,
> 
> patch is applied, now netisr is not eating CPU. but performance drops
> around 0.2Gbps according to previous kernel.
> 
> I tried also both net.isr.maxthreads=1 and net.isr.maxthreads=4 . results
> are same
> 
> Results are:
> 
> - with CCR - 1.8Gbps
> top:
>   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
>    14 root        -16    -     0B    16K CPU5     5   1:38 100.00% [crypto
> returns 8]
>     3 root        -16    -     0B    16K CPU1     1   0:58  77.83% [crypto
> returns 0]

Could you also try this patch?  It won't help with aesni.

diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c
index dfd22662e87f..bc23056e86ad 100644
--- a/sys/opencrypto/crypto.c
+++ b/sys/opencrypto/crypto.c
@@ -94,6 +94,7 @@ struct crypto_session {
 	void *softc;
 	uint32_t hid;
 	uint32_t capabilities;
+	uint64_t id;
 };
 
 SDT_PROVIDER_DEFINE(opencrypto);
@@ -572,6 +573,7 @@ crypto_select_driver(const struct cryptoini *cri, int flags)
 int
 crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid)
 {
+	static uint64_t sessid = 0;
 	crypto_session_t res;
 	void *softc_mem;
 	struct cryptocap *cap;
@@ -616,6 +618,7 @@ crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid)
 	softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO);
 	res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO);
 	res->softc = softc_mem;
+	res->id = atomic_fetchadd_64(&sessid, 1);
 
 	CRYPTO_DRIVER_LOCK();
 	cap = crypto_checkdriver(hid);
@@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp)
 		binuptime(&crp->crp_tstamp);
 #endif
 
-	crp->crp_retw_id = ((uintptr_t)crp->crp_session) % crypto_workers_num;
+	crp->crp_retw_id = crp->crp_session->id % crypto_workers_num;
 
 	if (CRYPTOP_ASYNC(crp)) {
 		if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {