From nobody Wed Mar 16 23:48:29 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 816DF1A162D0; Wed, 16 Mar 2022 23:48:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KJn653HKkz4n5J; Wed, 16 Mar 2022 23:48:29 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647474509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=vEqZUcxShkslayBd6rlmQhRDvQK9cPqa/jurz3dq62w=; b=GbWuHVRP3J1ek+05lWSw7vDc6zKQEnbIpWvhXM53JprqQCsSrg/lrsjLedJ17ELPG1Vyxq kUMlUz1GJXRNzf5CG+kbtVByYLpwfn9ZB1lfu1ymE/bR2C5rHXeUxOknewFq1fehP9BKP5 OBmHwnwe7sNac5IDZ6XxFXZe6SlA79eFaY+yl1zWi/s+0EGbtb4CoFWNJl21j9ki3EDTou oMKvvM2UHnPoCY0P+CKh5BilXiUk3JZjxL0P/ZjSqUs736V6quoJwGtKi/shqSTypns0Mh GXwrR19lhF3r3tNu29XlFH3mloJCSDBqNsM90z2+XHE4lDaH4IA11iZT2r4caA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 507201FBAA; Wed, 16 Mar 2022 23:48:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 22GNmTjq061593; Wed, 16 Mar 2022 23:48:29 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 22GNmTRq061592; Wed, 16 Mar 2022 23:48:29 GMT (envelope-from git) Date: Wed, 16 Mar 2022 23:48:29 GMT Message-Id: <202203162348.22GNmTRq061592@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kristof Provost Subject: git: 1ddb6c5518e0 - releng/13.1 - if_epair: fix race condition on multi-core systems List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kp X-Git-Repository: src X-Git-Refname: refs/heads/releng/13.1 X-Git-Reftype: branch X-Git-Commit: 1ddb6c5518e02953f3fe37d4da27a15ebd7a5a12 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647474509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=vEqZUcxShkslayBd6rlmQhRDvQK9cPqa/jurz3dq62w=; b=BroXGpVO6nFXgmMmtFFkCK69DtEmQQUTEscoOAr9K2nBTggDhxO4k/uLfqBokDIWXvj9xY 9FGdhVQGlZsYAcg+LQbbQeR9yjOciAfqQXIIBH/gcoV0tFmY6rMmzEso+aCAEZPb8K5Du+ g33vuBkaCSL/SA5FKzSGvkbssQogRabYLtWlFaURlhKHMvwQ7vKy/2Z0ArfSydWokS7ZE6 l1Uo0guaD/RWdQLpBQNpsPVPvbBytZBbdnROcgEUg8JXnYtNAzDkKHHY8Kv1vMd9aoaBU4 +4D07KfpXrAHoEPZarrdota+WGj4DL8bZz7sDn/lIEFD5nB/End5ER3HKdbJMg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1647474509; a=rsa-sha256; cv=none; b=opAHK78SvEZMjQ+Mp1D7BmT8mqvx6zmevFIdn2/96zjzQyJsMJGbJwIYSOYaQQ8KE6zN8T H1vygpcbp1HtIFFjJKtf3EiAzxdQPq+EfB5w3RHFHkmXvXJxMrSLsNkXEwi8dK5XAyh9Md 5rYkXpt66hQTgC2b8UZd1NmpM/MuL3cJopPUlG7Fep9wgIMnOvvYxDJXEjQ5tndocwXpYV nA7HyoYOYPcccXqzFqOFfNNVodMm4Ld+5TyW2Do7My4JEUO1tYY7Q1Ta5qPxMe6gJdV8HB kjI09bwlEHAsgrUcejO7quXMtk7VWTxr+FRRwAF/PAyoCUQuYV5OkWyjgC8XFA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch releng/13.1 has been updated by kp: URL: https://cgit.FreeBSD.org/src/commit/?id=1ddb6c5518e02953f3fe37d4da27a15ebd7a5a12 commit 1ddb6c5518e02953f3fe37d4da27a15ebd7a5a12 Author: Michael Gmelin AuthorDate: 2022-03-16 22:08:55 +0000 Commit: Kristof Provost CommitDate: 2022-03-16 23:40:40 +0000 if_epair: fix race condition on multi-core systems As an unwanted side effect of the performance improvements in 24f0bfbad57b9, epair interfaces stop forwarding traffic on higher load levels when running on multi-core systems. This happens due to a race condition in the logic that decides when to place work in the task queue(s) responsible for processing the content of ring buffers. In order to fix this, a field named state is added to the epair_queue structure. This field is used by the affected functions to signal each other that something happened in the underlying ring buffers that might require work to be scheduled in task queue(s), replacing the existing logic, which relied on checking if ring buffers are empty or not. epair_menq() does: - set BIT_MBUF_QUEUED - queue mbuf - if testandset BIT_QUEUE_TASK: enqueue task epair_tx_start_deferred() does: - swap ring buffers - process mbufs - clear BIT_QUEUE_TASK - if testandclear BIT_MBUF_QUEUED enqueue task PR: 262571 Approved by: re (gjb, early MFC) Reported by: Johan Hendriks MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D34569 (cherry picked from commit 66acf7685bcd8cf23b6c658a991637238a01859e) (cherry picked from commit bb9ad300f029d57c84804702daa2652542a2535f) --- sys/net/if_epair.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/sys/net/if_epair.c b/sys/net/if_epair.c index 818f25f0cdb5..16e7f602edee 100644 --- a/sys/net/if_epair.c +++ b/sys/net/if_epair.c @@ -98,11 +98,15 @@ static unsigned int next_index = 0; #define EPAIR_LOCK() mtx_lock(&epair_n_index_mtx) #define EPAIR_UNLOCK() mtx_unlock(&epair_n_index_mtx) +#define BIT_QUEUE_TASK 0 +#define BIT_MBUF_QUEUED 1 + struct epair_softc; struct epair_queue { int id; struct buf_ring *rxring[2]; volatile int ridx; /* 0 || 1 */ + volatile int state; /* taskqueue coordination */ struct task tx_task; struct epair_softc *sc; }; @@ -171,7 +175,8 @@ epair_tx_start_deferred(void *arg, int pending) } while (!atomic_fcmpset_int(&q->ridx, &ridx, nidx)); epair_if_input(sc, q, ridx); - if (! buf_ring_empty(q->rxring[nidx])) + atomic_clear_int(&q->state, (1 << BIT_QUEUE_TASK)); + if (atomic_testandclear_int(&q->state, BIT_MBUF_QUEUED)) taskqueue_enqueue(epair_tasks.tq[q->id], &q->tx_task); if_rele(sc->ifp); @@ -186,7 +191,6 @@ epair_menq(struct mbuf *m, struct epair_softc *osc) short mflags; struct epair_queue *q = NULL; uint32_t bucket; - bool was_empty; #ifdef RSS struct ether_header *eh; #endif @@ -234,14 +238,14 @@ epair_menq(struct mbuf *m, struct epair_softc *osc) #endif q = &osc->queues[bucket]; + atomic_set_int(&q->state, (1 << BIT_MBUF_QUEUED)); ridx = atomic_load_int(&q->ridx); - was_empty = buf_ring_empty(q->rxring[ridx]); ret = buf_ring_enqueue(q->rxring[ridx], m); if (ret != 0) { /* Ring is full. */ if_inc_counter(ifp, IFCOUNTER_OQDROPS, 1); m_freem(m); - goto done; + return (0); } if_inc_counter(ifp, IFCOUNTER_OPACKETS, 1); @@ -256,8 +260,7 @@ epair_menq(struct mbuf *m, struct epair_softc *osc) /* Someone else received the packet. */ if_inc_counter(oifp, IFCOUNTER_IPACKETS, 1); -done: - if (was_empty) + if (!atomic_testandset_int(&q->state, BIT_QUEUE_TASK)) taskqueue_enqueue(epair_tasks.tq[bucket], &q->tx_task); return (0); @@ -538,6 +541,7 @@ epair_clone_create(struct if_clone *ifc, char *name, size_t len, caddr_t params) q->rxring[0] = buf_ring_alloc(RXRSIZE, M_EPAIR, M_WAITOK, NULL); q->rxring[1] = buf_ring_alloc(RXRSIZE, M_EPAIR, M_WAITOK, NULL); q->ridx = 0; + q->state = 0; q->sc = sca; NET_TASK_INIT(&q->tx_task, 0, epair_tx_start_deferred, q); } @@ -560,6 +564,7 @@ epair_clone_create(struct if_clone *ifc, char *name, size_t len, caddr_t params) q->rxring[0] = buf_ring_alloc(RXRSIZE, M_EPAIR, M_WAITOK, NULL); q->rxring[1] = buf_ring_alloc(RXRSIZE, M_EPAIR, M_WAITOK, NULL); q->ridx = 0; + q->state = 0; q->sc = scb; NET_TASK_INIT(&q->tx_task, 0, epair_tx_start_deferred, q); }