From nobody Fri Aug 18 07:41:49 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RRv2j3jTsz4qrFc; Fri, 18 Aug 2023 07:41:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RRv2j28Nzz3G7q; Fri, 18 Aug 2023 07:41:49 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692344509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/+mkzK61223kZRJvmZqq/HdmSXHVJlhtXHUIe9ZTfRg=; b=GmeLiIxAt2EVQZPsIA2t1al1/gtR+Oxx2NYZYSm6orM5NCqhsX9lcbYurXTZb6mUjhIwc8 1MB0svYoeNw05HEXciFulWlNI3ncA09wY4CGfOmOs7VjT8JWaSLu5nsAv5ViGOciVuAmPT 18rnE3jZuLQ6MFSdJXQQu2cBtgGfvr2GGwhJruQDOloWINdpeXkP5Ufw/cDH0ovGA8AhCM nlshikUrrQCBBU+lD7kdg8mZMUXjsvgRmxBRDkb6vhxVhZL2iYP/h5RVjMIOeAaUWQ4HFm JX+w97pwrFrWaU9VtxFWnd+Ee+kwqLjyBjtvzqdzgDlP9RViR2CpUiw05iVgLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692344509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/+mkzK61223kZRJvmZqq/HdmSXHVJlhtXHUIe9ZTfRg=; b=NTHwVOjWkhZHsDs7Jfr4cQI8bHm4c6sc/XU/7h3WN1qAiZ90XTpKtR61FYnVYSrfrUQDxV Kwkirk8j9f/6CZKg4MKV6TqbCX8f3E7A1Y3hNaNs/ftSMPUWSN0asbTBsXX73EzLHSaBLP DoqAR3O6ucoJ2fabN9a8lcwZ5cjq1wKxu5kpXWYpxQSgRZ8Z1Oj3ICthR/guyU8hUiTBCv bpsFMC/7SgBtiAoDliV0EmCC4+djoY9X8G1h9Y3bZO+ftSaULLdYFJbhEqINu9k6v7oVNZ hL8GvINiXQUgFAsl/+cAiQRCeC8pfkWZFqDM0zBmKogoop21Dk1oJkF6lPMx5A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692344509; a=rsa-sha256; cv=none; b=eTiwW69d+BhbMaw1tdA4FkxuI+Wvh6KQPpWWsttG/3Awi4hgi0E6tCc7IfMTc3yBdTRJ6b perhthTRoe70PQpRy9MVSz0qj5YKy46dKnXpfCHCtmI3nJ0GZyBJYXkYvaPrAq45hoQWk9 2KJMdxYhEB1CXsFy08LUGXIHlk6Ws2Ug6iB70HoDiA3/8T2IyV1Iwr8/Waw3/ZjUg2NqOR BPXkb9zq+7zdX+zTZD203ybccTUoS6h6JN1HHpPv+7rlgkpvv6TLk2ftS/dDG93B9Im+lh zzOnYCzHRwaUDsslRCQcA1oJPOk6izF0jtpZ/Y2poSgd01pQkkJx0HLioqDyzw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RRv2j0ybfz16wF; Fri, 18 Aug 2023 07:41:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 37I7fnOt066782; Fri, 18 Aug 2023 07:41:49 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 37I7fnmO066779; Fri, 18 Aug 2023 07:41:49 GMT (envelope-from git) Date: Fri, 18 Aug 2023 07:41:49 GMT Message-Id: <202308180741.37I7fnmO066779@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Corvin =?utf-8?Q?K=C3=B6hne?= Subject: git: aa598e677a7c - stable/13 - bhyve: allocate guest memory for graphics stolen memory List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: corvink X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: aa598e677a7c3e6cf8e8524e9e8015df9c28a9f9 Auto-Submitted: auto-generated The branch stable/13 has been updated by corvink: URL: https://cgit.FreeBSD.org/src/commit/?id=aa598e677a7c3e6cf8e8524e9e8015df9c28a9f9 commit aa598e677a7c3e6cf8e8524e9e8015df9c28a9f9 Author: Corvin Köhne AuthorDate: 2023-05-11 09:10:07 +0000 Commit: Corvin Köhne CommitDate: 2023-08-18 07:32:07 +0000 bhyve: allocate guest memory for graphics stolen memory The graphics stolen memory is only GPU accessible. So, we don't have to copy any data to it as the guest will be unable to access it anyway. We just have to allocate and reserve some memory. That's done by adding an E820 entry for the graphics stolen memory. The guest firmware will pick up the E820 and reserve this range. Note that we try to reuse the host address as Intel states that newer Tiger Lake platforms need this [1]. [1] https://github.com/projectacrn/acrn-hypervisor/blob/e28d6fbfdfd556ff1bc3ff330e41d4ddbaa0f897/devicemodel/hw/pci/passthrough.c#L626-L629 Reviewed by: markj MFC after: 1 week Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D40059 (cherry picked from commit 1bd361eea28ae74b42d49bfc96b109a7eae9f3d3) --- usr.sbin/bhyve/pci_gvt-d.c | 64 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 62 insertions(+), 2 deletions(-) diff --git a/usr.sbin/bhyve/pci_gvt-d.c b/usr.sbin/bhyve/pci_gvt-d.c index 0be740921c92..9c5b0906ec29 100644 --- a/usr.sbin/bhyve/pci_gvt-d.c +++ b/usr.sbin/bhyve/pci_gvt-d.c @@ -8,17 +8,24 @@ #include #include -#include - #include +#include #include +#include "e820.h" #include "pci_gvt-d-opregion.h" #include "pci_passthru.h" +#define KB (1024UL) +#define MB (1024 * KB) +#define GB (1024 * MB) + #define PCI_VENDOR_INTEL 0x8086 +#define PCIM_BDSM_GSM_ALIGNMENT \ + 0x00100000 /* Graphics Stolen Memory is 1 MB aligned */ + #define GVT_D_MAP_GSM 0 static int @@ -41,6 +48,27 @@ gvt_d_probe(struct pci_devinst *const pi) return (0); } +static vm_paddr_t +gvt_d_alloc_mmio_memory(const vm_paddr_t host_address, const vm_paddr_t length, + const vm_paddr_t alignment, const enum e820_memory_type type) +{ + vm_paddr_t address; + + /* Try to reuse host address. */ + address = e820_alloc(host_address, length, E820_ALIGNMENT_NONE, type, + E820_ALLOCATE_SPECIFIC); + if (address != 0) { + return (address); + } + + /* + * We're not able to reuse the host address. Fall back to the highest usable + * address below 4 GB. + */ + return ( + e820_alloc(4 * GB, length, alignment, type, E820_ALLOCATE_HIGHEST)); +} + /* * Note that the graphics stolen memory is somehow confusing. On the one hand * the Intel Open Source HD Graphics Programmers' Reference Manual states that @@ -95,6 +123,38 @@ gvt_d_setup_gsm(struct pci_devinst *const pi) return (-1); } gsm->hva = NULL; /* unused */ + gsm->gva = NULL; /* unused */ + gsm->gpa = gvt_d_alloc_mmio_memory(gsm->hpa, gsm->len, + PCIM_BDSM_GSM_ALIGNMENT, E820_TYPE_RESERVED); + if (gsm->gpa == 0) { + warnx( + "%s: Unable to add Graphics Stolen Memory to E820 table (hpa 0x%lx len 0x%lx)", + __func__, gsm->hpa, gsm->len); + e820_dump_table(); + return (-1); + } + if (gsm->gpa != gsm->hpa) { + /* + * ACRN source code implies that graphics driver for newer Intel + * platforms like Tiger Lake will read the Graphics Stolen Memory + * address from an MMIO register. We have three options to solve this + * issue: + * 1. Patch the value in the MMIO register + * This could have unintended side effects. Without any + * documentation how this register is used by the GPU, don't do + * it. + * 2. Trap the MMIO register + * It's not possible to trap a single MMIO register. We need to + * trap a whole page. Trapping a bunch of MMIO register could + * degrade the performance noticeably. We have to test it. + * 3. Use an 1:1 host to guest mapping + * Maybe not always possible. As far as we know, no supported + * platform requires a 1:1 mapping. For that reason, just log a + * warning. + */ + warnx( + "Warning: Unable to reuse host address of Graphics Stolen Memory. GPU passthrough might not work properly."); + } return (0); }