git: 98d788c867b9 - main - opencrypto: Fix assignment of crypto completions to worker threads

[Mark Johnston]

The branch main has been updated by markj:

URL: https://cgit.FreeBSD.org/src/commit/?id=98d788c867b9e1d7a7e290254443b87ea77d8ab1

commit 98d788c867b9e1d7a7e290254443b87ea77d8ab1
Author:     Mark Johnston <markj at FreeBSD.org>
AuthorDate: 2021-01-20 01:34:35 +0000
Commit:     Mark Johnston <markj at FreeBSD.org>
CommitDate: 2021-01-20 01:34:35 +0000

    opencrypto: Fix assignment of crypto completions to worker threads
    
    Since r336439 we simply take the session pointer value mod the number of
    worker threads (ncpu by default).  On small systems this ends up
    funneling all completion work through a single thread, which becomes a
    bottleneck when processing IPSec traffic using hardware crypto drivers.
    (Software drivers such as aesni(4) are unaffected since they invoke
    completion handlers synchonously.)
    
    Instead, maintain an incrementing counter with a unique value per
    session, and use that to distribute work to completion threads.
    
    Reviewed by:    cem, jhb
    MFC after:      2 weeks
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D28159
---
 sys/opencrypto/crypto.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c
index 7bc230140568..fc4169fb2365 100644
--- a/sys/opencrypto/crypto.c
+++ b/sys/opencrypto/crypto.c
@@ -135,6 +135,7 @@ static	int crypto_drivers_size = 0;
 struct crypto_session {
 	struct cryptocap *cap;
 	struct crypto_session_params csp;
+	uint64_t id;
 };
 
 /*
@@ -908,6 +909,7 @@ int
 crypto_newsession(crypto_session_t *cses,
     const struct crypto_session_params *csp, int crid)
 {
+	static uint64_t sessid = 0;
 	crypto_session_t res;
 	struct cryptocap *cap;
 	int err;
@@ -944,6 +946,7 @@ crypto_newsession(crypto_session_t *cses,
 	    M_WAITOK | M_ZERO);
 	res->cap = cap;
 	res->csp = *csp;
+	res->id = atomic_fetchadd_64(&sessid, 1);
 
 	/* Call the driver initialization routine. */
 	err = CRYPTODEV_NEWSESSION(cap->cc_dev, res, csp);
@@ -1416,7 +1419,7 @@ crypto_dispatch(struct cryptop *crp)
 
 	CRYPTOSTAT_INC(cs_ops);
 
-	crp->crp_retw_id = ((uintptr_t)crp->crp_session) % crypto_workers_num;
+	crp->crp_retw_id = crp->crp_session->id % crypto_workers_num;
 
 	if (CRYPTOP_ASYNC(crp)) {
 		if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {