From nobody Mon Aug 21 19:54:01 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV3892PKrz4rFYN; Mon, 21 Aug 2023 19:54:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV3891wX1z4Wlf; Mon, 21 Aug 2023 19:54:01 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692647641; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=JHwfvKeRSFa0xWve3RqGM3TsGMKpwXxsow60VYV3XdE=; b=k58b4R/jmmi4wKKDuYk0vE79Hr32Y4T6cNWB1g5Hk8uJTwXVpVgG01vsB6E/361JFCk/nq ulcYOhpzboeKXKimm+7UjeljS8qo7uQybuLpv92hW2KNuEDbvOqOT1teGgTVQr4amjG+9+ PCNYaBHCK9kK+Uu7s0uUOKyzEVOG1eZFVWRpRPSY7RU1yCKB1K63RemTr/FcQ051X3HIJd byUW4L8+IicFz+x+FPXDhQkzTiqqIayPirKGtHbr5XxR3MHwxhjN7Qsx69d1AhDPqXUtJq z7r3jf9d/Bh57QdmZPJjnh0UYJXoZrDrh9PDXlwaBcbxoctDu7LcwZ2RI4TqYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692647641; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=JHwfvKeRSFa0xWve3RqGM3TsGMKpwXxsow60VYV3XdE=; b=okwoXlYQosw+IlC9Y1LZ3ExXQHsmOdPqMVoWYjSvVjto2/m8Fdz95/WsDsdFEuNwkBzksq wsOPPa0k5A8c3yQfUovRX/YI0O9B3XID4w/wObNfl6Mo+xlVSz6K7LeFy1/1vaLxYaqatQ IVe4ekcF1KoaN6jrcXi80NkQVAePpKp5yC+el60thReZBr01bfCMB7Clh1woNnn70oJosB cqDxQU+VXPuCXEZn8PHT+00BXgJj5VN4LxaHer2U7ZnvrGsZhFJEvqzcf48O7VBci1rism xISc1EKq09crOxKZNL0Pew6kzkqVep/k3ftAe/y3dAhkrcVEMLm68o2yMUnF4Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692647641; a=rsa-sha256; cv=none; b=bUa/SD3CGPjod8l5Fi2zkHI6evs235wnta9OKJaIhSRJb0RkeM+28SxwNUUhoYXWNppkEE FaetAcaQ/klmtrbXFIccdfsG8dIrJh9hPDV4AoXrnwvirksYLAZ220+5GgWVuL5/ohqwdm 6D0VQdepcK+IzZfrGEpIH5TrrAQqKzIejoLPbAzwkuZxS1cBJdWZ3OwjKmicMDi8YdPSRj ve6tJb11HA4LCUIlZTLmFMpoLdE5wgMdXl/qzogwmf9tBr1tyNZn5FcxMlcs2UFqOnBtjd BoQGlSldBpcl+Jnq7Elg5wSP+ne/msKq79A8of/vCSL0sS0Mnvanb76FwI99wQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RV389101szZV6; Mon, 21 Aug 2023 19:54:01 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 37LJs1Ew079856; Mon, 21 Aug 2023 19:54:01 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 37LJs153079853; Mon, 21 Aug 2023 19:54:01 GMT (envelope-from git) Date: Mon, 21 Aug 2023 19:54:01 GMT Message-Id: <202308211954.37LJs153079853@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Ed Maste Subject: git: 4258eb5a0d97 - main - x86: handle domains with no CPUs usable for intr delivery List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: emaste X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 4258eb5a0d971cf9b1ea5e8e98535e29ef3053f1 Auto-Submitted: auto-generated The branch main has been updated by emaste: URL: https://cgit.FreeBSD.org/src/commit/?id=4258eb5a0d971cf9b1ea5e8e98535e29ef3053f1 commit 4258eb5a0d971cf9b1ea5e8e98535e29ef3053f1 Author: Ed Maste AuthorDate: 2023-08-18 03:29:33 +0000 Commit: Ed Maste CommitDate: 2023-08-21 19:52:10 +0000 x86: handle domains with no CPUs usable for intr delivery We can end up with a domain having no CPUs capable of receiving I/O interrupts. This can occur, for example, when all APIC IDs in a given domain are 256 or greater, and we have no IOMMU. In this case disable per-domain interrupt support, effectively reverting to the behaviour before commit a48de40bcc09 ("Only use CPUs in the domain the device is attached to for default"). This has a performance impact but at least allows the system to be functional. It is a stop- gap until we can rely on the presence of an IOMMU on all x86 platforms. Thanks to AMD for providing the high-thread-count machine I used for testing this change, and to cperciva for testing on other hardware. Reviewed by: jhb Tested by: cperciva, emaste Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D41501 --- sys/x86/x86/intr_machdep.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/sys/x86/x86/intr_machdep.c b/sys/x86/x86/intr_machdep.c index b8dbe3611a42..b43fa790d264 100644 --- a/sys/x86/x86/intr_machdep.c +++ b/sys/x86/x86/intr_machdep.c @@ -578,10 +578,17 @@ DB_SHOW_COMMAND(irqs, db_show_irqs) /* * Support for balancing interrupt sources across CPUs. For now we just * allocate CPUs round-robin. + * + * XXX If the system has a domain with without any usable CPUs (e.g., where all + * APIC IDs are 256 or greater and we do not have an IOMMU) we use + * intr_no_domain to fall back to assigning interrupts without regard for + * domain. Once we can rely on the presence of an IOMMU on all x86 platforms + * we can revert this. */ cpuset_t intr_cpus = CPUSET_T_INITIALIZER(0x1); static int current_cpu[MAXMEMDOM]; +static bool intr_no_domain; static void intr_init_cpus(void) @@ -589,7 +596,15 @@ intr_init_cpus(void) int i; for (i = 0; i < vm_ndomains; i++) { + if (CPU_OVERLAP(&cpuset_domain[i], &intr_cpus) == 0) { + intr_no_domain = true; + printf("%s: unable to route interrupts to CPUs in domain %d\n", + __func__, i); + } + current_cpu[i] = 0; + if (intr_no_domain && i > 0) + continue; if (!CPU_ISSET(current_cpu[i], &intr_cpus) || !CPU_ISSET(current_cpu[i], &cpuset_domain[i])) intr_next_cpu(i); @@ -615,6 +630,8 @@ intr_next_cpu(int domain) return (PCPU_GET(apic_id)); #endif + if (intr_no_domain) + domain = 0; mtx_lock_spin(&icu_lock); apic_id = cpu_apic_ids[current_cpu[domain]]; do { @@ -622,7 +639,8 @@ intr_next_cpu(int domain) if (current_cpu[domain] > mp_maxid) current_cpu[domain] = 0; } while (!CPU_ISSET(current_cpu[domain], &intr_cpus) || - !CPU_ISSET(current_cpu[domain], &cpuset_domain[domain])); + (!CPU_ISSET(current_cpu[domain], &cpuset_domain[domain]) && + !intr_no_domain)); mtx_unlock_spin(&icu_lock); return (apic_id); }