From nobody Fri Jun 16 06:14:15 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qj84m0W4jz4fGbb; Fri, 16 Jun 2023 06:14:16 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qj84m00Vsz3qb5; Fri, 16 Jun 2023 06:14:16 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686896056; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WQ93rTByRAvXF9WXABTeFbLKf1HFHnW2+Mk2sLJMvrg=; b=iKoYjeJnN+wH3xagmA1r99TgfmzZAO02gIXmll9hrAg83MZ2+cxIVa+ph3Gx4/y5UmdkON bEl0rY4Hbk4LhLgIUDBJh4s13IQOdG1qgyy26szCept6UAFM+Htk0jFK3j/vfmU/Dcoq3P XO7uEGI4U7smKjeEGiZnY2nWo3rcfwUveYIaALyclsCMK7MrVh5GkOrBGuMI6rXk3lhI+4 j79zYmNB3djdK9Rga/PNGhKkiY4oZkxJGg+XSZY9dVvF2KTnZXfcwGvoJFiw68FBZkMLVE PO985DA+OaSmZRzPbfY6EkEAxRU4KkAXgGhQtY6bal+oCb8Fh/bPEVyzegGvmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686896056; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WQ93rTByRAvXF9WXABTeFbLKf1HFHnW2+Mk2sLJMvrg=; b=vgmEyb7+LQJ27XAHeeSfO+DqfKYXKRSNDLuH8aIweOBfHcgWx0K8MoS+6zpX90kjGFtkZe LKiomcQWogX6dQ2sQ/rY7bvRuFEFS6GTdrIN5BALIcNLV+IFs29Om6wR4lzyZ6HiiwptsV Js/CGWIGTOo+byQIxcRCiZ4sts4NjuqVEpQa1mh1Uawi6Vp4TfkpC7LK/uxFAQybDlTJt8 B0+g+mtV3LKUTwx+YzWowRkpcKokutp87/qzyX7X4fB4nF5VKgfhkMzwaeLaoiUZ5ZMOs8 ykBewvlbdSrIBMWt+vaBhfi7vRfp6O5vFagOnwCi9ifpS+RuxeKQhSpTySwmGw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1686896056; a=rsa-sha256; cv=none; b=tWqvXrr2HyKhI1df7HD+oUxPrTqwKCAgT98bd1cg8DtZ8lyEhC0DVDeeyyNo64ymvmoBIL usFw8tBxl4gZ/RSz+o3zjIcOfVVxwBWvxSZI0B3qL7IsvL9Ak7Jk58EaLyxwsmZc2F6Fra CfsJy9fnIhJhRFy5neCMAGRJTnT6KiXoUDLP/L88vjcUweuUd4HFrGFkOO2elK1KyFrnZo eMSrEUaXAj01pKXGxsbcZyyZjJbDrIY/wnWiJhfWaClFALy3b+6W1iclILmupGArsM3LPL YSZgQMCnx2CYGERiS4nKW0KF8FFsvfs5YmHmG3KzbIHASTm6pmpSoj7EUzNPNw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Qj84l69j1zGy5; Fri, 16 Jun 2023 06:14:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 35G6EFUg035149; Fri, 16 Jun 2023 06:14:15 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 35G6EFJp035148; Fri, 16 Jun 2023 06:14:15 GMT (envelope-from git) Date: Fri, 16 Jun 2023 06:14:15 GMT Message-Id: <202306160614.35G6EFJp035148@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: =?utf-8?Q?Corvin=20K=C3=B6hne?= Subject: git: 1bd361eea28a - main - bhyve: allocate guest memory for graphics stolen memory List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: corvink X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1bd361eea28ae74b42d49bfc96b109a7eae9f3d3 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by corvink: URL: https://cgit.FreeBSD.org/src/commit/?id=1bd361eea28ae74b42d49bfc96b109a7eae9f3d3 commit 1bd361eea28ae74b42d49bfc96b109a7eae9f3d3 Author: Corvin Köhne AuthorDate: 2023-05-11 09:10:07 +0000 Commit: Corvin Köhne CommitDate: 2023-06-16 05:54:02 +0000 bhyve: allocate guest memory for graphics stolen memory The graphics stolen memory is only GPU accessible. So, we don't have to copy any data to it as the guest will be unable to access it anyway. We just have to allocate and reserve some memory. That's done by adding an E820 entry for the graphics stolen memory. The guest firmware will pick up the E820 and reserve this range. Note that we try to reuse the host address as Intel states that newer Tiger Lake platforms need this [1]. [1] https://github.com/projectacrn/acrn-hypervisor/blob/e28d6fbfdfd556ff1bc3ff330e41d4ddbaa0f897/devicemodel/hw/pci/passthrough.c#L626-L629 Reviewed by: markj MFC after: 1 week Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D40059 --- usr.sbin/bhyve/pci_gvt-d.c | 64 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 62 insertions(+), 2 deletions(-) diff --git a/usr.sbin/bhyve/pci_gvt-d.c b/usr.sbin/bhyve/pci_gvt-d.c index 0be740921c92..9c5b0906ec29 100644 --- a/usr.sbin/bhyve/pci_gvt-d.c +++ b/usr.sbin/bhyve/pci_gvt-d.c @@ -8,17 +8,24 @@ #include #include -#include - #include +#include #include +#include "e820.h" #include "pci_gvt-d-opregion.h" #include "pci_passthru.h" +#define KB (1024UL) +#define MB (1024 * KB) +#define GB (1024 * MB) + #define PCI_VENDOR_INTEL 0x8086 +#define PCIM_BDSM_GSM_ALIGNMENT \ + 0x00100000 /* Graphics Stolen Memory is 1 MB aligned */ + #define GVT_D_MAP_GSM 0 static int @@ -41,6 +48,27 @@ gvt_d_probe(struct pci_devinst *const pi) return (0); } +static vm_paddr_t +gvt_d_alloc_mmio_memory(const vm_paddr_t host_address, const vm_paddr_t length, + const vm_paddr_t alignment, const enum e820_memory_type type) +{ + vm_paddr_t address; + + /* Try to reuse host address. */ + address = e820_alloc(host_address, length, E820_ALIGNMENT_NONE, type, + E820_ALLOCATE_SPECIFIC); + if (address != 0) { + return (address); + } + + /* + * We're not able to reuse the host address. Fall back to the highest usable + * address below 4 GB. + */ + return ( + e820_alloc(4 * GB, length, alignment, type, E820_ALLOCATE_HIGHEST)); +} + /* * Note that the graphics stolen memory is somehow confusing. On the one hand * the Intel Open Source HD Graphics Programmers' Reference Manual states that @@ -95,6 +123,38 @@ gvt_d_setup_gsm(struct pci_devinst *const pi) return (-1); } gsm->hva = NULL; /* unused */ + gsm->gva = NULL; /* unused */ + gsm->gpa = gvt_d_alloc_mmio_memory(gsm->hpa, gsm->len, + PCIM_BDSM_GSM_ALIGNMENT, E820_TYPE_RESERVED); + if (gsm->gpa == 0) { + warnx( + "%s: Unable to add Graphics Stolen Memory to E820 table (hpa 0x%lx len 0x%lx)", + __func__, gsm->hpa, gsm->len); + e820_dump_table(); + return (-1); + } + if (gsm->gpa != gsm->hpa) { + /* + * ACRN source code implies that graphics driver for newer Intel + * platforms like Tiger Lake will read the Graphics Stolen Memory + * address from an MMIO register. We have three options to solve this + * issue: + * 1. Patch the value in the MMIO register + * This could have unintended side effects. Without any + * documentation how this register is used by the GPU, don't do + * it. + * 2. Trap the MMIO register + * It's not possible to trap a single MMIO register. We need to + * trap a whole page. Trapping a bunch of MMIO register could + * degrade the performance noticeably. We have to test it. + * 3. Use an 1:1 host to guest mapping + * Maybe not always possible. As far as we know, no supported + * platform requires a 1:1 mapping. For that reason, just log a + * warning. + */ + warnx( + "Warning: Unable to reuse host address of Graphics Stolen Memory. GPU passthrough might not work properly."); + } return (0); }