From nobody Fri May 10 09:30:08 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VbNrw5pDdz5J2dj; Fri, 10 May 2024 09:30:08 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VbNrw4PYrz4Fqs; Fri, 10 May 2024 09:30:08 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1715333408; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cDclrGGpo1kz7Iz8KUL9zoIpLhG4gna1L/2dO1qdfg0=; b=GLlx028OtQ2aBICgH2+pB4WPmMKz6FqG/kGBJwpKf803QFvqsiNB6MUE6DeDJPWfTchClY 4CuaUK72ZO+DUiGXiCETWmsen2HrwnJyBAMOxX53IMDX6+M1lDO639gOJx6T8OicHsgbkf x8ImJFFyrtES4c3S4ZV7w5TCcEnesv9ov8L6IYah3Zuou5lpf82y/KNISZSOLGO6Y/Rc7Q Maue0ngLDRS6tvLDjbliyjplSUraoh0Jsil6hNN8CUnkENndPsfZgJB55lmT0xu0+Aw34X p4h6CkEVg0k/23vvnny3pStR0n8bLTvrWp/m2ylPeKQN99LllO0bPRTHaeIJWA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1715333408; a=rsa-sha256; cv=none; b=hjl/EcvXi6UNSkOtghOp+Xaol2b4rY5w4hf+1ZbgjfmiM7hnO4+Z8tzwPWR/NkWwvCDwD8 /oHBTM34F6ACF7rKmkk9wvW565sMRiAITNFDoeHg5vL5QDE6SYj00MXujxZKSHbgWplXtM n1slG7ktdvZPxInz5lfIY95BqbSg0MvX/SPm9/6n45w1Uasvcd7xWYQP4EG4GCCSSCyKfJ 6W5X+kRz5I2cUpli/AuK5is0+/5kGCzRUDEFa+Nsiv5feI/TzahX2v6tzAQBq31zxI95bi OlPoj7ONlMa/DM0fqQCcuwISLC8lM2TCssDDtBqX008pOXmTPQkpa1qKbSsUVw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1715333408; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cDclrGGpo1kz7Iz8KUL9zoIpLhG4gna1L/2dO1qdfg0=; b=lvD53MWfKtddvKBLiKj1qHbCETeld0ffwa71dcDnFFMsxq6leBvmF3y8paRp70GD4bVPUB +Kh1HrtNV92bLKHdHSfI/wjNe9nSOYVliRtx0zLTgpZtk3zw8a9GgBlJMvDp/t63Aaaous EXj0tvQXK0RpFj2+36yMgyigbfaPXtIr5tYL0EeXhpCCVOVpAN3alnVTmhk3LxsW0v/uLx /3IXtWOvi3/zur32RKtARxlYDph0YcaDwYKTUjHoswo1OobZ50XXL8e0k3K67HG5bj9a3F UG5Znyr5bBK6mxRPufRi59XO6givZt5m+EclF10KtsHoDCbA+Uj682n8p41Aug== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VbNrw410rzXqF; Fri, 10 May 2024 09:30:08 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 44A9U8e8084511; Fri, 10 May 2024 09:30:08 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 44A9U8BQ084506; Fri, 10 May 2024 09:30:08 GMT (envelope-from git) Date: Fri, 10 May 2024 09:30:08 GMT Message-Id: <202405100930.44A9U8BQ084506@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Andrew Turner Subject: git: c78ebc69c2aa - main - arm64: Support a shared release for spin-table List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: andrew X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: c78ebc69c2aa5454b4dc8fd7451b3b0d031205b8 Auto-Submitted: auto-generated The branch main has been updated by andrew: URL: https://cgit.FreeBSD.org/src/commit/?id=c78ebc69c2aa5454b4dc8fd7451b3b0d031205b8 commit c78ebc69c2aa5454b4dc8fd7451b3b0d031205b8 Author: Andrew Turner AuthorDate: 2024-04-29 14:23:42 +0000 Commit: Andrew Turner CommitDate: 2024-05-10 09:29:24 +0000 arm64: Support a shared release for spin-table When releasing multiple CPUs that share a release address we need them to wait for their turn to boot. Add a mechanism to do this by booting them until they enable the TLB before waiting their turn to enter init_secondary. Reviewed by: jhibbits, kevans Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45082 --- sys/arm64/arm64/locore.S | 55 ++++++++++++++++++++++++++++++++++++++++---- sys/arm64/arm64/mp_machdep.c | 21 +++++++++++++---- 2 files changed, 67 insertions(+), 9 deletions(-) diff --git a/sys/arm64/arm64/locore.S b/sys/arm64/arm64/locore.S index fd77938edae9..94a50b735bc6 100644 --- a/sys/arm64/arm64/locore.S +++ b/sys/arm64/arm64/locore.S @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -192,12 +193,50 @@ END(_start) #ifdef SMP /* - * mpentry(unsigned long) + * void + * mpentry_psci(unsigned long) * - * Called by a core when it is being brought online. + * Called by a core when it is being brought online with psci. * The data in x0 is passed straight to init_secondary. */ -ENTRY(mpentry) +ENTRY(mpentry_psci) + mov x26, xzr + b mpentry_common +END(mpentry_psci) + +/* + * void + * mpentry_spintable(void) + * + * Called by a core when it is being brought online with a spin-table. + * Reads the new CPU ID and passes this to init_secondary. + */ +ENTRY(mpentry_spintable) + ldr x26, =spintable_wait + b mpentry_common +END(mpentry_spintable) + +/* Wait for the current CPU to be released */ +LENTRY(spintable_wait) + /* Read the affinity bits from mpidr_el1 */ + mrs x1, mpidr_el1 + ldr x2, =CPU_AFF_MASK + and x1, x1, x2 + + adrp x2, ap_cpuid +1: + ldr x0, [x2, :lo12:ap_cpuid] + cmp x0, x1 + b.ne 1b + + str xzr, [x2, :lo12:ap_cpuid] + dsb sy + sev + + ret +LEND(mpentry_spintable) + +LENTRY(mpentry_common) /* Disable interrupts */ msr daifset, #DAIF_INTR @@ -228,6 +267,14 @@ ENTRY(mpentry) mp_virtdone: BTI_J + /* + * Allow this CPU to wait until the kernel is ready for it, + * e.g. with spin-table but each CPU uses the same release address + */ + cbz x26, 1f + blr x26 +1: + /* Start using the AP boot stack */ adrp x4, bootstack ldr x4, [x4, :lo12:bootstack] @@ -258,7 +305,7 @@ mp_virtdone: msr tpidr_el1, x18 b init_secondary -END(mpentry) +LEND(mpentry_common) #endif /* diff --git a/sys/arm64/arm64/mp_machdep.c b/sys/arm64/arm64/mp_machdep.c index 9c6175445572..bd13dde9cee0 100644 --- a/sys/arm64/arm64/mp_machdep.c +++ b/sys/arm64/arm64/mp_machdep.c @@ -106,7 +106,8 @@ static void ipi_stop(void *); static u_int fdt_cpuid; #endif -void mpentry(unsigned long cpuid); +void mpentry_psci(unsigned long cpuid); +void mpentry_spintable(void); void init_secondary(uint64_t); /* Synchronize AP startup. */ @@ -114,6 +115,7 @@ static struct mtx ap_boot_mtx; /* Used to initialize the PCPU ahead of calling init_secondary(). */ void *bootpcpu; +uint64_t ap_cpuid; /* Stacks for AP initialization, discarded once idle threads are started. */ void *bootstack; @@ -420,7 +422,10 @@ enable_cpu_spin(uint64_t cpu, vm_paddr_t entry, vm_paddr_t release_paddr) { vm_paddr_t *release_addr; - release_addr = pmap_mapdev(release_paddr, sizeof(*release_addr)); + ap_cpuid = cpu & CPU_AFF_MASK; + + release_addr = pmap_mapdev_attr(release_paddr, sizeof(*release_addr), + VM_MEMATTR_DEFAULT); if (release_addr == NULL) return (ENOMEM); @@ -432,6 +437,10 @@ enable_cpu_spin(uint64_t cpu, vm_paddr_t entry, vm_paddr_t release_paddr) "sev \n" ::: "memory"); + /* Wait for the target CPU to start */ + while (atomic_load_64(&ap_cpuid) != 0) + __asm __volatile("wfe"); + return (0); } @@ -475,7 +484,6 @@ start_cpu(u_int cpuid, uint64_t target_cpu, int domain, vm_paddr_t release_addr) bootstack = (char *)bootstacks[cpuid] + MP_BOOTSTACK_SIZE; printf("Starting CPU %u (%lx)\n", cpuid, target_cpu); - pa = pmap_extract(kernel_pmap, (vm_offset_t)mpentry); /* * A limited set of hardware we support can only do spintables and @@ -483,10 +491,13 @@ start_cpu(u_int cpuid, uint64_t target_cpu, int domain, vm_paddr_t release_addr) * PSCI branch here. */ MPASS(release_addr == 0 || !psci_present); - if (release_addr != 0) + if (release_addr != 0) { + pa = pmap_extract(kernel_pmap, (vm_offset_t)mpentry_spintable); err = enable_cpu_spin(target_cpu, pa, release_addr); - else + } else { + pa = pmap_extract(kernel_pmap, (vm_offset_t)mpentry_psci); err = enable_cpu_psci(target_cpu, pa, cpuid); + } if (err != 0) { pcpu_destroy(pcpup);