From nobody Tue Jan 21 23:15:50 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Yd33V2jLNz5lk4p; Tue, 21 Jan 2025 23:15:50 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Yd33V22P9z3J1Y; Tue, 21 Jan 2025 23:15:50 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1737501350; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fYhT1diWU0VyJpemecRy8fRy2Qi21gyn7Koi/h0WKTQ=; b=Yg70DsIjFu/aeSd453zVW2Z+ces1pHYp8+pPCaoR4eu+my4KMl+fx6o68epPg/qNxmwIC8 MNoha3zlnx+N503kSZ1NZTrTGuaVhCHerfLBercPPl0ZxNfCwgT3SYOh5rwDUQ0rHnY/+m YckeC1JDyKNz3DhCtEfYEbEcNOV7N/IMPtgjotPI8Mqq9e55OP0v6jV3d5fMvK2+A4p2/O DT7W5XxaYqGznkJmi6vYWoeY2wIdwGLUo42gra1fYsT1IQvJECKp02VFYooQKVjnLA7uwm 5PG1jzAqMgr7mHOKrl+8/M1+7xmr525Snf+vKHLG/McsmM9jzRv9RmJ9E8B72A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1737501350; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fYhT1diWU0VyJpemecRy8fRy2Qi21gyn7Koi/h0WKTQ=; b=T8Oq2U8cIYNXAbcpztlOqTpwcsrTfIJ3okZU4LWcOVAQFFlZOyUmUQk93/ytRSfb08V8Uc r/HY7BXlxF7+mkCiEGlCa0fw13wTrG2s4b2SRwI4sKMh8laQXokWhcvm6TYOwqKFbMHIRU CirQ9CmVGgEpqA0CZULg/s5LTEohiEQpL5tos6K8bhMC7I7n8CLi3lPGZ+lbgbsrVMu6RG Myo4+AENe5dg3q9SZPtlwPdCAlv0ynQzOWiaFEDa/Srv+FgXPpoEbS68rmtLKKbgj0LARm ce0ukA7FED6rCVT79jry1sjNsiFg/LTHOivwPJrYA9Jf9NyosnKi1+fYAAv4oA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1737501350; a=rsa-sha256; cv=none; b=UYvkJCEqv9A+pYIbR2VAAfgw9+EFIata0MgF4udzGQUM72HpPIFH1ruPNfpzUV0+Xx/1D/ AoUNVaYQCHYj9yVxGDhOYF3efypdaHcX6KoPCUnqk+xefiwQyVhgFltJ5OaFZDYXyDLT7l Gz5rHRtXr4Yf7hdEPStF1MbUoMh1H1Vyp6Qjz0damS92jZ+TBOUpUsNIsHVihLGAIq8FFs PfRsGq4r8dFre7IsXmqL+18zBbuAZzTE+13cJ8/gdLCF4hLjYO50gMCl1d5pQOjKuyqoeL VBhWI2LaGUpSj3+6iNMKH7nI9mAEQN69xuPgPGCRZyf+LXdMbJDyN25krkx5Yg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Yd33V0rngzBvh; Tue, 21 Jan 2025 23:15:50 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 50LNFo0B078567; Tue, 21 Jan 2025 23:15:50 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 50LNFo5Z078564; Tue, 21 Jan 2025 23:15:50 GMT (envelope-from git) Date: Tue, 21 Jan 2025 23:15:50 GMT Message-Id: <202501212315.50LNFo5Z078564@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Justin Hibbits Subject: git: 6f0c2938e895 - main - Fix latency spikes on return from stop on POWER9+ List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhibbits X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6f0c2938e895bd3ddb14f44acf36dba3c6156a3f Auto-Submitted: auto-generated The branch main has been updated by jhibbits: URL: https://cgit.FreeBSD.org/src/commit/?id=6f0c2938e895bd3ddb14f44acf36dba3c6156a3f commit 6f0c2938e895bd3ddb14f44acf36dba3c6156a3f Author: Timothy Pearson AuthorDate: 2025-01-20 20:03:40 +0000 Commit: Justin Hibbits CommitDate: 2025-01-21 23:14:41 +0000 Fix latency spikes on return from stop on POWER9+ On POWER9 and higher platforms, cpu_idle_power9() directly control the external interrupt enable lines when entering / exiting stop states. This does not provide needed information to the rest of the kernel regarding the core going into a stop state, and results in random, significant latency spikes (>200ms) due to the stopped core not receiving wakeup interrupts in a deterministic manner. Bring cpu_idle_power9() in line with cpu_idle_powerx() by using spinlock entry / exit to control the interrupt state vs. direct MSR read / write. Signed-off-by: Timothy Pearson --- sys/powerpc/powerpc/cpu.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/sys/powerpc/powerpc/cpu.c b/sys/powerpc/powerpc/cpu.c index 4b8da50cf642..37eb01e2b74f 100644 --- a/sys/powerpc/powerpc/cpu.c +++ b/sys/powerpc/powerpc/cpu.c @@ -855,7 +855,6 @@ cpu_idle_powerx(sbintime_t sbt) static void cpu_idle_power9(sbintime_t sbt) { - register_t msr; int max_stop_state = cpu_idle_max_stop_state; /* Limit maximum stop state to valid values */ @@ -877,20 +876,30 @@ cpu_idle_power9(sbintime_t sbt) cpu_idle_max_stop_state = max_stop_state; } - msr = mfmsr(); - /* Suspend external interrupts until stop instruction completes. */ - mtmsr(msr & ~PSL_EE); + /* + * Enter spinlock and suspend external interrupts until the stop + * instruction completes. + */ + spinlock_enter(); + + /* Final scheduler checks before core shutdown */ + if (sched_runnable()) { + /* Exit spinlock and re-enable external interrupts */ + spinlock_exit(); + return; + } + /* Set the stop state to lowest latency, wake up to next instruction */ - /* Set maximum transition level to 2, for deepest lossless sleep. */ - mtspr(SPR_PSSCR, (2 << PSSCR_MTL_S) | (0 << PSSCR_RL_S)); - /* "stop" instruction (PowerISA 3.0) */ + mtspr(SPR_PSSCR, (max_stop_state << PSSCR_MTL_S) | (0 << PSSCR_RL_S)); + + /* Shut down core using "stop" instruction (PowerISA 3.0) */ __asm __volatile (".long 0x4c0002e4"); + /* * Re-enable external interrupts to capture the interrupt that caused - * the wake up. + * the wake up. Exit spinlock. */ - mtmsr(msr); - + spinlock_exit(); } #endif