From nobody Thu Dec 22 09:22:47 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nd4bW1zm6z1Gn97; Thu, 22 Dec 2022 09:22:47 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nd4bW1YXdz43Rr; Thu, 22 Dec 2022 09:22:47 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671700967; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/6EYOO3t7qijJ6wxlhv/ux2SU2fiSMvtvoaqTeDEltI=; b=aJoeSTrXDNbPaQlxcSqfaY1VeikCkZWvNsmFaU/O/0/j0Z+oOSsFnl+7XY78omSotF+394 1NBrJfpjb3Kexp0ohMoIkl9XCrI7MXHC2o+AHAAU63HC7iTBy/KOHJrpTKQ6PVDXK94cXB jeuZobBd0zhJQmINzBwqQFT9glxayT948R/c7VwrGh+ZJm3aM8x75RtYMH+ctJD+OA6H6l +EiSEwJ6Uz0sN8RUzf2xkhJy7gJNLp8MWpa7ON2Yf53gWan/goCfhLmKjQhxxTsnFfU5Js Bi/KzJacYXfCs6fwbmtk8//rOlnkXt6sTDXz8AthA57/VcKYppqzuyq9J1DzRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671700967; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/6EYOO3t7qijJ6wxlhv/ux2SU2fiSMvtvoaqTeDEltI=; b=CuiHnFpgNlBzaLBQ4HuEU44xoHMjK8Qc8ZjwnoYxLroMxiq2Euz2YVRSPBv9cPx4bw/H99 eA3wlN61Lv+X0e8Wam5jJffcH26lcuxVSxVH79NGNkxnV2931bsGzN4JetLTxGSNQOlYRx cOyKMMT54a7r9C0jO7MF2wfjdSYmfKq4BGLrNHpI0QHIbfYYcYS3bzqVrP59Qc6upWZeNM 5nZZSQhfPtkPMehaqwhEUKim31J7v7f86K9jCzCswBMqAUbpl+/YaXWwvvAm7Yob+DgA9/ wPLcwpJ3WmKg+vOcDaYAXeTaVvrOzT9gDD4sHmi0PC5iaGq8TUHyXGQBmiu2/A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671700967; a=rsa-sha256; cv=none; b=Gzi4d1Jw3OzeLVQ7AOAhulJFhEBV7FOF9KUUsH7ucisNN1h8/WH4n4cMTFZ8PP+GZ/zKuB tafHIABH4WlZMLMlYRz97JLoYSeBWhX7GzzB46JF6aD3qErTdoX1PiqAsqzUE8cqSjAFBx neVaGS15Gdo+PbtUKGnEBvPkcs2553+ShiE5DyFfa3F+8w9n5upmWtz9dmpGWorM2kz5p8 QPU77at396/onLmrNzT6S4GPWC5ZvdoD7odfz3ej414rtuj2YWlOdtpGQkZ3Aim4LBngLZ hl94OoOofRPItt49deGQGwn+JN3LlhEb+biVKyOOZ/wsXpt48/5hWIdsw2AJkg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Nd4bW0VKTzq8n; Thu, 22 Dec 2022 09:22:47 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 2BM9Mlc3038723; Thu, 22 Dec 2022 09:22:47 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 2BM9Ml6O038722; Thu, 22 Dec 2022 09:22:47 GMT (envelope-from git) Date: Thu, 22 Dec 2022 09:22:47 GMT Message-Id: <202212220922.2BM9Ml6O038722@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: =?utf-8?Q?Corvin=20K=C3=B6hne?= Subject: git: 0eebb2d30a34 - stable/13 - vmm: Fix AP startup with old userspace binaries. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: corvink X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 0eebb2d30a34ac48254fc54f38cbfe13441782c7 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by corvink: URL: https://cgit.FreeBSD.org/src/commit/?id=0eebb2d30a34ac48254fc54f38cbfe13441782c7 commit 0eebb2d30a34ac48254fc54f38cbfe13441782c7 Author: John Baldwin AuthorDate: 2022-10-26 21:22:56 +0000 Commit: Corvin Köhne CommitDate: 2022-12-22 09:12:22 +0000 vmm: Fix AP startup with old userspace binaries. Older binaries that do not request IPI exits to userspace do not start user threads for other vCPUs until a STARTUP IPI triggers a VM_EXITCODE_SPINUP_AP exit to userland. This means that those vcpus are not yet active (in terms of vm_active_cpus) when the INIT and STARTUP IPIs are delivered to the vCPUs. The changes in commit 0bda8d3e9f7a changed the INIT and STARTUP IPIs to reuse the existing vlapic_calcdest() function. This function silently ignores IPIs sent to inactive vCPUs. As a result, when using an old bhyve binary, the INIT and STARTUP IPIs sent to wakeup APs were ignored. To fix, restructure the compat code for the INIT and STARTUP IPIs to ignore the results of vlapic_calcdest() and manually parse the APIC ID and resulting vcpuid. As part of this, make the compat code always conditonal on the ipi_exit capability being disabled. Reviewed by: c.koehne_beckhoff.com, markj Differential Revision: https://reviews.freebsd.org/D37093 (cherry picked from commit 769b884e2e2eb84d25eca2a5462ae0a6c4dcd2a7) --- sys/amd64/vmm/io/vlapic.c | 63 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 18 deletions(-) diff --git a/sys/amd64/vmm/io/vlapic.c b/sys/amd64/vmm/io/vlapic.c index e1a57c4abcfc..5deca142c5d1 100644 --- a/sys/amd64/vmm/io/vlapic.c +++ b/sys/amd64/vmm/io/vlapic.c @@ -1119,20 +1119,61 @@ vlapic_icrlo_write_handler(struct vlapic *vlapic, bool *retu) break; case APIC_DELMODE_INIT: - CPU_FOREACH_ISSET(i, &dmask) { + if (!vlapic->ipi_exit) { + if (!phys) + break; + + i = vm_apicid2vcpuid(vlapic->vm, dest); + if (i >= vm_get_maxcpus(vlapic->vm) || + i == vlapic->vcpuid) + break; + /* - * Userland which doesn't support the IPI exit requires - * that the boot state is set to SIPI here. + * Userland which doesn't support the IPI exit + * requires that the boot state is set to SIPI + * here. */ vlapic2 = vm_lapic(vlapic->vm, i); vlapic2->boot_state = BS_SIPI; - CPU_SET(i, &ipimask); + break; } + CPU_COPY(&dmask, &ipimask); break; case APIC_DELMODE_STARTUP: + if (!vlapic->ipi_exit) { + if (!phys) + break; + + /* + * Old bhyve versions don't support the IPI + * exit. Translate it into the old style. + */ + i = vm_apicid2vcpuid(vlapic->vm, dest); + if (i >= vm_get_maxcpus(vlapic->vm) || + i == vlapic->vcpuid) + break; + + /* + * Ignore SIPIs in any state other than wait-for-SIPI + */ + vlapic2 = vm_lapic(vlapic->vm, i); + if (vlapic2->boot_state != BS_SIPI) + break; + vlapic2->boot_state = BS_RUNNING; + + vmexit = vm_exitinfo(vlapic->vm, vlapic->vcpuid); + vmexit->exitcode = VM_EXITCODE_SPINUP_AP; + vmexit->u.spinup_ap.vcpu = i; + vmexit->u.spinup_ap.rip = vec << PAGE_SHIFT; + + *retu = true; + break; + } + CPU_FOREACH_ISSET(i, &dmask) { vlapic2 = vm_lapic(vlapic->vm, i); + /* * Ignore SIPIs in any state other than wait-for-SIPI */ @@ -1155,20 +1196,6 @@ vlapic_icrlo_write_handler(struct vlapic *vlapic, bool *retu) vmexit->u.ipi.dmask = dmask; *retu = true; - - /* - * Old bhyve versions don't support the IPI exit. Translate it - * into the old style. - */ - if (!vlapic->ipi_exit) { - if (mode == APIC_DELMODE_STARTUP) { - vmexit->exitcode = VM_EXITCODE_SPINUP_AP; - vmexit->u.spinup_ap.vcpu = CPU_FFS(&ipimask) - 1; - vmexit->u.spinup_ap.rip = vec << PAGE_SHIFT; - } else { - *retu = false; - } - } } return (0);