git: 10ed8ab4ab9b - stable/12 - opencrypto: Fix assignment of crypto completions to worker threads

[Mark Johnston]

The branch stable/12 has been updated by markj:

URL: https://cgit.FreeBSD.org/src/commit/?id=10ed8ab4ab9bd0239f2913ac2f35af9b9f76221d

commit 10ed8ab4ab9bd0239f2913ac2f35af9b9f76221d
Author:     Mark Johnston <markj at FreeBSD.org>
AuthorDate: 2021-01-20 01:34:35 +0000
Commit:     Mark Johnston <markj at FreeBSD.org>
CommitDate: 2021-02-03 14:37:17 +0000

    opencrypto: Fix assignment of crypto completions to worker threads
    
    Since r336439 we simply take the session pointer value mod the number of
    worker threads (ncpu by default).  On small systems this ends up
    funneling all completion work through a single thread, which becomes a
    bottleneck when processing IPSec traffic using hardware crypto drivers.
    (Software drivers such as aesni(4) are unaffected since they invoke
    completion handlers synchonously.)
    
    Instead, maintain an incrementing counter with a unique value per
    session, and use that to distribute work to completion threads.
    
    Reviewed by:    cem, jhb
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D28159
    
    (cherry picked from commit 98d788c867b9e1d7a7e290254443b87ea77d8ab1)
---
 sys/opencrypto/crypto.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c
index dfd22662e87f..bc23056e86ad 100644
--- a/sys/opencrypto/crypto.c
+++ b/sys/opencrypto/crypto.c
@@ -94,6 +94,7 @@ struct crypto_session {
 	void *softc;
 	uint32_t hid;
 	uint32_t capabilities;
+	uint64_t id;
 };
 
 SDT_PROVIDER_DEFINE(opencrypto);
@@ -572,6 +573,7 @@ again:
 int
 crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid)
 {
+	static uint64_t sessid = 0;
 	crypto_session_t res;
 	void *softc_mem;
 	struct cryptocap *cap;
@@ -616,6 +618,7 @@ restart:
 	softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO);
 	res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO);
 	res->softc = softc_mem;
+	res->id = atomic_fetchadd_64(&sessid, 1);
 
 	CRYPTO_DRIVER_LOCK();
 	cap = crypto_checkdriver(hid);
@@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp)
 		binuptime(&crp->crp_tstamp);
 #endif
 
-	crp->crp_retw_id = ((uintptr_t)crp->crp_session) % crypto_workers_num;
+	crp->crp_retw_id = crp->crp_session->id % crypto_workers_num;
 
 	if (CRYPTOP_ASYNC(crp)) {
 		if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {