From nobody Sat Jan 04 20:33:28 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YQXGl1VkYz5jm9P for ; Sat, 04 Jan 2025 20:34:07 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YQXGk4j6Kz4Jpq for ; Sat, 4 Jan 2025 20:34:06 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-2efb17478adso18656092a91.1 for ; Sat, 04 Jan 2025 12:34:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736022845; x=1736627645; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=6vgnVg8yWzmQdgzdllVbbwlR3h75s5wGa5WsT09ioys=; b=ffD9GmPW/0ZMUpLfAbw382nV3wTgYIuo6l7UvHzAAiQzLuK7bXKh8oZ7Hnirr8UYyM H4sc9uMZTnX0dUa5eWiUQujH02z2abSQuCn6t3Mv0XBonNLvsBNhWVGnTn5zqrf2QB0p qmIEPZIjJFUqjk6M5cm156hyc3UP4c156nDjGcDSkOPWCbyYAE/S1wNFciCf9z6x3T1a yFTrmWEv8IjVfFXvn+fhIljSwHJegh6lStN0by9Y1gHVpsDoYrsmLJt63c6xFE0QLIM1 d2Qk+JSMGKHrhvsX8/RoFY82Ep5yJWG74fNhwLGGcLfPi2p2OcoBbeQHkEEBafj1zlUD ZVTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736022845; x=1736627645; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=6vgnVg8yWzmQdgzdllVbbwlR3h75s5wGa5WsT09ioys=; b=nF+mJDn5tRgA6Cji3ShtujsduRJ6S0aZjb2vscyq+u+znq2n1nNUFXh7vl/SoM1dre ODnLZ7YX06Znk12Q5a5IJ9aoC/mLBiuQZKWwErqPxS4pHc3mj4MY4E0f31zbLeU7V124 PMs3uCUXmY00B/SqfJR7vy2q3zQZZgdBjtwjE0rtUeoJ+fShDmdee8/cWBA2NGf+wxhx tXt9Q663PF772voqaE7FE9N0hfbgpXikuusxdocU5t+xzg1OqPRBtiMKx1s/mfrWU3K4 MWqVKjcv4HT793yVpRPQ/coQ2bJqcLSPgjbj3zQCbHEC9aT1LHhbbtQcEY4Kmo8YSvGO sESQ== X-Gm-Message-State: AOJu0YyQQzGxSDewQ9cJgpbSffe1YBIMn+ne2mCBSos4V1vvXttv9OrM sQNZTH24vBVOhQh41dh3T6kdjiU8mbWGKk7N449QuTl/S+0IvihbPLY1PVDNmHKIHOElGiP7oIR Z4gHhYNNhou5FXiBkeKfVG8CavZ7KgSo1sVA= X-Gm-Gg: ASbGnct//dTFZdQeFmmTnqEII05g/KCKtmscA/H8XqMvvr4cniQFmMBusiAZqKwTJ7L S3xWbHDc5R37czzfOlXVuKLbaT8H9+CCCyL/+juw= X-Google-Smtp-Source: AGHT+IHBgr2sdWfGAAmBrxSO89gMVF1Luv4aIFWUgo/PhhWPmlBjXbCD36mTkVvW0ewzMq6Ug+zDN1c6omfg0msddPI= X-Received: by 2002:a17:90b:1346:b0:2ee:d024:e4fc with SMTP id 98e67ed59e1d1-2f452eeda2amr82188152a91.33.1736022845150; Sat, 04 Jan 2025 12:34:05 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Sat, 4 Jan 2025 21:33:28 +0100 Message-ID: Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? To: Peter Wood Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="00000000000001dfe1062ae751d0" X-Rspamd-Queue-Id: 4YQXGk4j6Kz4Jpq X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --00000000000001dfe1062ae751d0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Peter, Please make sure your GPU is booted in UEFI and not CSM mode. This can be changed in the host BIOS. Cheers. On Sat, Jan 4, 2025 at 9:24=E2=80=AFPM Peter Wood wrot= e: > Happy new year all. > > I've been using bhyve happily for a year or two now, multiple machines > with VT-d running happily passing SAS cards and USB cards into VMs. > > I've reached the point where I want to pass a GPU in for accelerated > encoding/decoding/etc (scrypted, tdarr, jellyfin). I picked up an Intel A= RC > A380, as it's encoders/decoders are pretty decent for my use case - I'd > also seen that there had been success with people using the iGPU's in Int= el > CPUs successfully. > > Unfortunately after attaching the GPUs PCI device to ppt, and attempting > to start a VM with it attached, I'm greated by an error that seems to foc= us > on iGPUs? > > /usr/sbin/bhyve -A -H -w -u -S -c 2 -m 8G -l com1,/dev/nmdm202B -l > bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/usr/local/var/cache= /bmd/k8s-worker-3.vars > -s 0,hostbridge -s 1,lpc -s 2,nvme,/dev/zvol/vm/k8s-worker-3 -s > 3,ahci-cd,/mnt/vm/isos/ubuntu-24.04-live-server-amd64.iso -s > 5,virtio-net,tap9 -s 4,passthru,4/0/0 -s 6,fbuf,tcp=3D0.0.0.0:6007,w=3D12= 80,h=3D720,vga=3Dio > -s 7,xhci,tablet test > bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory. > GPU passthrough might not work properly. > bhyve: gvt_d_setup_opregion: Invalid OpRegion signature > bhyve: gvt_d_init: Unable to setup OpRegion > Device emulation initialization error: No such file or directory > > The machine is an AMD EPYC 7343 on a ASRock ROMED8-2T, it has a dedicated > onboard GPU (attached to the BMC), which the BIOS is configured to > encourage operating systems to use as the primary display - and sure enou= gh > the loader and BSD console are presented through the BMCs ASPEED AST2500. > > Digging through the freebsd source tree, it appears that that pci_gvt-d.c > is responsible for this, if it's Intel and a Display then attempt the set > up of graphics memory. > > https://github.com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6cb79075e183= 0b97594bef3/usr.sbin/bhyve/amd64/pci_gvt-d.c#L44 > > Has anyone experimented with this? Can I just patch this out, rebuild > bhyve and expect a chance of success? > > Cheers, > > P. > -- > *Peter Wood* > peter@alastria.net > > --=20 Mario. --00000000000001dfe1062ae751d0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hi Peter,

=C2=A0

Please make sure your GPU is booted in UEFI and not = CSM mode. This can be changed in the host B= IOS.

Cheers.



On Sat, Jan 4, 2025 at 9:24=E2=80=AFPM Peter Wood <peter@alastria.net> wrote:
H= appy new year all.

I've been using bhyve h= appily for a year or two now, multiple machines with VT-d running happily p= assing SAS cards and USB cards into VMs.

I've = reached the point where I want to pass a GPU in for accelerated encoding/de= coding/etc (scrypted, tdarr, jellyfin). I picked up an Intel ARC A380, as i= t's encoders/decoders are pretty decent for my use case - I'd also = seen that there had been success with people using the iGPU's in Intel = CPUs successfully.

Unfortunately after attaching t= he GPUs PCI device to ppt, and attempting to start a VM with it attached, I= 'm greated by an error that seems to focus on iGPUs?

=
/usr/sbin/bhyve -A -H -w -u -S -c 2 -m 8G -l com1,/dev/nmdm202B = -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/usr/local/var/cach= e/bmd/k8s-worker-3.vars -s 0,hostbridge -s 1,lpc -s 2,nvme,/dev/zvol/vm/k8s= -worker-3 -s 3,ahci-cd,/mnt/vm/isos/ubuntu-24.04-live-server-amd64.iso -s 5= ,virtio-net,tap9 -s 4,passthru,4/0/0 -s 6,fbuf,tcp=3D0.0.0.0:6007,w=3D1280,h=3D720,vga=3Dio -s 7= ,xhci,tablet test
bhyve: Warning: Unable to reuse host address of Graphi= cs Stolen Memory. GPU passthrough might not work properly.
bhyve: gvt_d_= setup_opregion: Invalid OpRegion signature
bhyve: gvt_d_init: Unable to = setup OpRegion
Device emulation initialization error: No such file or di= rectory

The machine is an AMD EPYC 7343 on a A= SRock ROMED8-2T, it has a dedicated onboard GPU (attached to the BMC), whic= h the BIOS is configured to encourage operating systems to use as the prima= ry display - and sure enough the loader and BSD console are presented throu= gh the BMCs ASPEED AST2500.

Digging through the freebsd source= tree, it appears that that pci_gvt-d.c is responsible for this, if it'= s Intel and a Display then attempt the set up of graphics memory.
https://github.com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6cb79075e18= 30b97594bef3/usr.sbin/bhyve/amd64/pci_gvt-d.c#L44

<= div>Has anyone experimented with this? Can I just patch this out, rebuild b= hyve and expect a chance of success?

Cheers,
=

P.
--
P= eter Wood



--
Ma= rio.
--00000000000001dfe1062ae751d0--