From nobody Mon Aug 01 14:19:12 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LxKxX5Qb2z4XtLD; Mon, 1 Aug 2022 14:19:12 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LxKxX4f27z48m0; Mon, 1 Aug 2022 14:19:12 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1659363552; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=YGd3YEpo2XrgIgy5qz6ptTM3QZHcHOR3JvKCGGa3vvk=; b=X17dGk6eyft/0NfJqIIXsY4RvisJi4gvv99ptTYTW9MS6Knq8OL0KjfSCA1bE9Bk9AiWqn PnvJq2hjEIZw9dFOmPcxEdSejEH0VQmLij33f84kIj/IbBDkt8r0KUUf8e6VMZeH+gtwRP LiBZbLNsoxVHwZL2bGKypmbKjxN3fL00q5uvFDILI8L9wURqHvsPCOqVrJtN9suOWi4uQg W5YuPzFxjJzKRS/gzHXGHOpWfUCbYD2YJkfwAp15pFHz4TkP0bH63XB5fVt7s0hYrRvYlV me4ttX0uZOuTxLaD1qqsJ+WmqFOVwmkrfOJE59I4EnZYJRgybw8rvDx7tkDJHw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4LxKxX3XbSzcFG; Mon, 1 Aug 2022 14:19:12 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 271EJCEA025186; Mon, 1 Aug 2022 14:19:12 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 271EJCX1025185; Mon, 1 Aug 2022 14:19:12 GMT (envelope-from git) Date: Mon, 1 Aug 2022 14:19:12 GMT Message-Id: <202208011419.271EJCX1025185@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 20abb8075086 - stable/13 - sched_ule: Enable preemption of curthread in the load balancer List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 20abb8075086dd31fae4ad39e84a44ce429a357e Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1659363552; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=YGd3YEpo2XrgIgy5qz6ptTM3QZHcHOR3JvKCGGa3vvk=; b=rNHoPEcGhZc3Fj1l35Q0wP8HQ0zkcVYvZ2IzSZlPWeNb+TwZVYa4Vr/rE8g9jK/SpOMywi eOkTIMRvvBGoB+Qze2QUd5bsZdfxrlPGE0yOTPUOEztRK9u7waaKJ5cXre2wqCXEeBSWfm HMB7DTT3fctrFU7XjvwohJ+xWdm1GKLLoWpNaYIwt4HQooIiWQ78v2+vGhT9D/Ny3x+1fT eXBL8QdyH7Q4fg2T3sG3RQ+DcJaJBCztppCMwLYrcoK8XMB+S7AJIQFbhZi7gCu83/EjVS XXE4XgVyudVvHhkhHTBpVNDI9J3duUOCfdMQFugxfVhAG/CvQdYEQNqgyWAnyw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1659363552; a=rsa-sha256; cv=none; b=brrfxj5OoTI0RvrcMn+Iox8vKrV+0K+QTrBh1vZmiGoL2w5aGgpa2X5ryL8XuF+oCWGsq7 By2RTXH8QO4cdH9wGdhJVRiXVgomccIaAnSmyI/xVZCgDyM6mRc1Yo0XGQoFtMeSN9Y0Ua nFK2oiIvHvy0dDoI+mIuR3DXeT0q/HwKGUwxvhNGg2cC/oWMjtmCd4KVfvzWc62C6Hiv2b HAf1kRA9o0JnGNBh0Ti1Bfn1WtaUX6gZjvRVBkWcEHfZ89bvZhI+oa6ge4sgdmZT7WDYcI dplHGKcWCI89DOCj+9SMB+lX40YvQajBRjkY7SxLfchj+0ZQgd1aGnLJJA+lqw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=20abb8075086dd31fae4ad39e84a44ce429a357e commit 20abb8075086dd31fae4ad39e84a44ce429a357e Author: Mark Johnston AuthorDate: 2022-07-14 14:23:43 +0000 Commit: Mark Johnston CommitDate: 2022-08-01 14:12:36 +0000 sched_ule: Enable preemption of curthread in the load balancer The load balancer executes from statclock and periodically tries to move threads among CPUs in order to balance load. It may move a thread to the current CPU (the loader balancer always runs on CPU 0). When it does so, it may need to schedule preemption of the interrupted thread. Use sched_setpreempt() to do so, same as sched_add(). PR: 264867 Reviewed by: mav, kib, jhb Sponsored by: The FreeBSD Foundation (cherry picked from commit 0927ff78147b4d00a75054bbad299946208e1e91) --- sys/kern/sched_ule.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/sys/kern/sched_ule.c b/sys/kern/sched_ule.c index 886f58c0dc84..dc5736b5a41a 100644 --- a/sys/kern/sched_ule.c +++ b/sys/kern/sched_ule.c @@ -307,6 +307,7 @@ static struct tdq tdq_cpu; #define TDQ_UNLOCK(t) mtx_unlock_spin(TDQ_LOCKPTR((t))) #define TDQ_LOCKPTR(t) ((struct mtx *)(&(t)->tdq_lock)) +static void sched_setpreempt(int); static void sched_priority(struct thread *); static void sched_thread_priority(struct thread *, u_char); static int sched_interact_score(struct thread *); @@ -949,13 +950,15 @@ sched_balance_pair(struct tdq *high, struct tdq *low) lowpri = tdq_move(high, low); if (lowpri != -1) { /* - * In case the target isn't the current cpu notify it of + * In case the target isn't the current CPU notify it of * the new load, possibly sending an IPI to force it to - * reschedule. + * reschedule. Otherwise maybe schedule a preemption. */ cpu = TDQ_ID(low); if (cpu != PCPU_GET(cpuid)) tdq_notify(low, lowpri); + else + sched_setpreempt(low->tdq_lowpri); ret = true; } } @@ -2632,20 +2635,19 @@ sched_choose(void) } /* - * Set owepreempt if necessary. Preemption never happens directly in ULE, - * we always request it once we exit a critical section. + * Set owepreempt if the currently running thread has lower priority than "pri". + * Preemption never happens directly in ULE, we always request it once we exit a + * critical section. */ -static inline void -sched_setpreempt(struct thread *td) +static void +sched_setpreempt(int pri) { struct thread *ctd; int cpri; - int pri; - - THREAD_LOCK_ASSERT(curthread, MA_OWNED); ctd = curthread; - pri = td->td_priority; + THREAD_LOCK_ASSERT(ctd, MA_OWNED); + cpri = ctd->td_priority; if (pri < cpri) ctd->td_flags |= TDF_NEEDRESCHED; @@ -2722,7 +2724,7 @@ sched_add(struct thread *td, int flags) if (cpu != PCPU_GET(cpuid)) tdq_notify(tdq, lowpri); else if (!(flags & SRQ_YIELDING)) - sched_setpreempt(td); + sched_setpreempt(td->td_priority); #else tdq = TDQ_SELF(); /* @@ -2738,7 +2740,7 @@ sched_add(struct thread *td, int flags) } (void)tdq_add(tdq, td, flags); if (!(flags & SRQ_YIELDING)) - sched_setpreempt(td); + sched_setpreempt(td->td_priority); #endif if (!(flags & SRQ_HOLDTD)) thread_unlock(td);