From nobody Tue Aug 29 17:25:54 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZvTZ5Czzz4s12F; Tue, 29 Aug 2023 17:25:54 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZvTZ4dRNz4FHc; Tue, 29 Aug 2023 17:25:54 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693329954; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=H8Vqq+pJw4SXiK1rWsCLaPpev03VvWVsc6VM909w/vI=; b=SIz2md+UM1c9u2WESmMwV6A6+vfBhY8X0LV1jlRcgj9vLt3zZycor00Yb0fvBoly+M2/Ay p4LUWFdrDyXDUsyZ8FKioddo5Nk2lTOeglMwzyC2MlrIm3Egm0nAAjv61sVxaemICZFnrp 3GALKoNEqC703OGTghxMZzoX1GT852cqWnKAp94/c8X2IpSx3k3hnCWxVu//c3gYGYC4Tv 8J3lfTd88/HDlL4cMfCFKRpRywk7F38jPxWbQsQH6r+QZUp5kSN4W5zSJwhRhbCPjoXyoE 5K91QKDaKahRFp1KjlkfbPvIs0iH84zPTqzBNqLLBmTDU/rzgHUMrVyt0jz7mw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693329954; a=rsa-sha256; cv=none; b=QL9PH4UZkKGyuGmZ/mFbv/eI6VMfzn2LpN0i7LjDgl1O36dhiWtDcRLb7vypAyLsUgvi3y 7yv3dixZJs4GrHORYRgS8NLOg8kUjB48L2FD271NQjzVVAdzzjt/un43CBlwhDsjUPtTlr EEnXzjojZBliKhjV+A4YCVrRbT+0BwRaBQJRoQRn3K6aXIwyA/A6kWvvVRmjxiRMXCEJ0p 8Z/V+3CTZ+mLfGX50/r18CWrt2SGpvGhjbn2Srbo1ag0E49vEhczIil8EtTYwd1rGcjMPh XCUKvwexeIImIECLCLm4zMMaUgudeOn4F9InOd0uoIZHXpsRftO7YT90E6FU2A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693329954; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=H8Vqq+pJw4SXiK1rWsCLaPpev03VvWVsc6VM909w/vI=; b=WvRQn0GIYPdTDzI4nAhx3KsKs05VtYnV+Pkcu1dadtcv/YO5k54vA64Kkl2HcNZ1Pw0cR6 wvLnDdJedj0swKmSwtBpzwgbJFMCiQjKI4Rvp0b/JVZ2crUAyNcr2aqy8IXmXAK5dagabp a1U+KgkZI7zsHyhyDRZhfTlTiTRFpQqCjKOgVE0UdTyOyKIP0YIpYcxhZb2M8zIty9GDes jBwQdFx0EIL3LWF+vN0FoJnVCcHm6lSqy9iGpjyInxwSgb33/N+8dyqxRQw7MrJlwQlvDY 92ZZZtQzx4BCnITtelk6me90wv3UlEsAC9NnJ9+jIGRDDykNyKpqr8QpPn9oBQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RZvTZ3jsQz1172; Tue, 29 Aug 2023 17:25:54 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 37THPsOW057205; Tue, 29 Aug 2023 17:25:54 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 37THPsL9057202; Tue, 29 Aug 2023 17:25:54 GMT (envelope-from git) Date: Tue, 29 Aug 2023 17:25:54 GMT Message-Id: <202308291725.37THPsL9057202@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Ed Maste Subject: git: 0b029e9e8594 - main - x86: Introduce APIC ID limit by default on AMD hardware List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: emaste X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 0b029e9e85943d565c72aa58353538aeac68aa36 Auto-Submitted: auto-generated The branch main has been updated by emaste: URL: https://cgit.FreeBSD.org/src/commit/?id=0b029e9e85943d565c72aa58353538aeac68aa36 commit 0b029e9e85943d565c72aa58353538aeac68aa36 Author: Ed Maste AuthorDate: 2023-08-18 03:39:08 +0000 Commit: Ed Maste CommitDate: 2023-08-29 17:25:30 +0000 x86: Introduce APIC ID limit by default on AMD hardware Lack of an AMD IOMMU driver means we cannot successfully route interrupts to APIC IDs 255 and over. Do not add the corresponding CPUs to the per-domain lists of CPUs to which interrupts can be assigned. This change should be reverted (or, at least the APIC ID limit) once we have an AMD IOMMU / interrupt remapping driver. See also commits fa5f94140a83 ("msi: handle error from BUS_REMAP_INTR in msi_assign_cpu") and 4258eb5a0d97 ("x86: handle domains with no CPUs usable for intr delivery."). Reviewed by: markj, jhb Tested by: cperciva (earlier version) MFC after: 3 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D41618 --- sys/x86/x86/mp_x86.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/sys/x86/x86/mp_x86.c b/sys/x86/x86/mp_x86.c index 76a1d14f371a..ddcb54b63d88 100644 --- a/sys/x86/x86/mp_x86.c +++ b/sys/x86/x86/mp_x86.c @@ -153,6 +153,11 @@ SYSCTL_INT(_machdep, OID_AUTO, hyperthreading_intr_allowed, CTLFLAG_RDTUN, &hyperthreading_intr_allowed, 0, "Allow interrupts on HTT logical CPUs"); +static int intr_apic_id_limit = -1; +SYSCTL_INT(_machdep, OID_AUTO, intr_apic_id_limit, CTLFLAG_RDTUN, + &intr_apic_id_limit, 0, + "Maximum permitted APIC ID for interrupt delivery (-1 is unlimited)"); + static struct topo_node topo_root; static int pkg_id_shift; @@ -258,6 +263,22 @@ topo_probe_amd(void) if ((amd_feature2 & AMDID2_CMP) == 0) return; + /* + * XXX Lack of an AMD IOMMU driver prevents use of APIC IDs above + * xAPIC_MAX_APIC_ID. This is a workaround so we boot and function on + * AMD systems with high thread counts, albeit with reduced interrupt + * performance. + * + * We should really set the limit to xAPIC_MAX_APIC_ID by default, and + * have the IOMMU driver increase it. That way if a driver is present + * but disabled, or is otherwise not able to route the interrupts, the + * system can fall back to a functional state. That will require a more + * substantial change though, including having the IOMMU initialize + * earlier. + */ + if (intr_apic_id_limit == -1) + intr_apic_id_limit = xAPIC_MAX_APIC_ID; + /* For families 10h and newer. */ pkg_id_shift = (cpu_procinfo2 & AMDID_COREID_SIZE) >> AMDID_COREID_SIZE_SHIFT; @@ -1170,6 +1191,8 @@ set_interrupt_apic_ids(void) continue; if (cpu_info[apic_id].cpu_disabled) continue; + if (intr_apic_id_limit >= 0 && apic_id > intr_apic_id_limit) + continue; /* Don't let hyperthreads service interrupts. */ if (cpu_info[apic_id].cpu_hyperthread &&