From nobody Sat Jan 04 20:23:55 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YQX3D71Bfz5jlyJ for ; Sat, 04 Jan 2025 20:24:08 +0000 (UTC) (envelope-from peter@alastria.net) Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YQX3C5Kgfz4Hn2 for ; Sat, 4 Jan 2025 20:24:07 +0000 (UTC) (envelope-from peter@alastria.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=alastria.net header.s=google header.b=Do7bsEeP; spf=pass (mx1.freebsd.org: domain of peter@alastria.net designates 2607:f8b0:4864:20::835 as permitted sender) smtp.mailfrom=peter@alastria.net; dmarc=none Received: by mail-qt1-x835.google.com with SMTP id d75a77b69052e-467b74a1754so17544431cf.1 for ; Sat, 04 Jan 2025 12:24:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alastria.net; s=google; t=1736022247; x=1736627047; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=9Em71hikgBb0Wjc3J2r0vmko7zA96ha9OmWakZyo1zk=; b=Do7bsEePqUDzhosuZuuMR+FJo0aO97hllEXj8a8FfMcue5NVzaCVccU6/RVlqomZ36 P6AInf+IRH1Ux1CkOHrJ/iIR5xPvX/MqzVZrt4qL8GQYiXCIKOaAT/xMhd99Q3TGlZrY 7URWEeoKugMuxmJ2l4/MrK44X2WH4A1E2S4Rj8Hd961AGf6bbJxY3dWMvbibstV6jGRj CV/4jff1VMfwvcmHJC6M+kv8JC2ASTYJKwOWCAYeTy+R87P6kVQk8WheMBpLWdg6LWRi WKP0bDXcqtunoXUZCF0Y9YuAPWj7Hc2nMnmpinnn/SKnzgQD6m84PBYhZ47uQFG64iWq vFJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736022247; x=1736627047; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=9Em71hikgBb0Wjc3J2r0vmko7zA96ha9OmWakZyo1zk=; b=n/lPaTbuWENSL5jQnYAoQFuS9PajlB15cim/CTNPwjjOOjp5p97MxyMtof2ogAoH2S gT4RtTbqfglk3AY78FUzIqTMsvH8a7VG3/sNVub0SevTljSAbxjdT9QwDtGzIqrNszCA N06uDxNzuF0dEALE8P2lZYVMTg0tgJeLR8K7CDsAuhJEwVm4cE1h/zyV0c4ueFqem1pc RuhPMDw6mb3lrSo8ND9jjGE4SB1un9u6CVgBzI/BTKjeLx49eEVq2HzqfhyQ7fZkd3WI yvUDRziKUuGO/t0CT3evp7PTqvxpjpxJuBSRktPaif3ARjp8RCsOW+nJgWjXkFPBq5lB Y5VA== X-Gm-Message-State: AOJu0YwTE9hwv2AYBOX7Z2rN+r1T/ThT9065XgNKU72Olv2T3piD84Wi ddRHqlgBnIn6HgSG8YT0udxpiUG4kAQ8jUcCVcKZyppvNIsOWdqyOjDpyx1Vb48AN1rTH5UMpKW kKVcdm0ovGIqGRDSUi+cVGVRMhTAQPinHnz6AS1Y4dLhtZXh6UZM= X-Gm-Gg: ASbGncsUK6zxkUPnom1qoP9NwZ2HGbuU/L5Eoy0h2M8EfI9vSkwbiyuJH+7ah9uBczh v18Z3fmGP/rZ3/6aDcmI2ybrUdeNsDU1Sk25A2ab1fe9nZpNat/oGP5IbcTlf4CSsxKhO X-Google-Smtp-Source: AGHT+IGiGdnPKnsme+zAkihTqsrauq1ZRISK7eIbAyXIFb3REjaY2ot63fhJ5AxIScmcCq3MT0w2O3Eju/WfNugU+1Q= X-Received: by 2002:ac8:5716:0:b0:467:6133:3372 with SMTP id d75a77b69052e-46a4a8f402emr878582231cf.25.1736022246691; Sat, 04 Jan 2025 12:24:06 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 From: Peter Wood Date: Sat, 4 Jan 2025 20:23:55 +0000 Message-ID: Subject: bhyve/passthru for Intel dGPU (ARC A380)? To: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000562b7a062ae72d63" X-Rspamd-Queue-Id: 4YQX3C5Kgfz4Hn2 X-Spamd-Bar: / X-Spamd-Result: default: False [-0.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; URI_COUNT_ODD(1.00)[5]; HTTP_TO_IP(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[alastria.net:s=google]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[alastria.net:+]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[alastria.net]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[peter]; RCVD_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::835:from] --000000000000562b7a062ae72d63 Content-Type: text/plain; charset="UTF-8" Happy new year all. I've been using bhyve happily for a year or two now, multiple machines with VT-d running happily passing SAS cards and USB cards into VMs. I've reached the point where I want to pass a GPU in for accelerated encoding/decoding/etc (scrypted, tdarr, jellyfin). I picked up an Intel ARC A380, as it's encoders/decoders are pretty decent for my use case - I'd also seen that there had been success with people using the iGPU's in Intel CPUs successfully. Unfortunately after attaching the GPUs PCI device to ppt, and attempting to start a VM with it attached, I'm greated by an error that seems to focus on iGPUs? /usr/sbin/bhyve -A -H -w -u -S -c 2 -m 8G -l com1,/dev/nmdm202B -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/usr/local/var/cache/bmd/k8s-worker-3.vars -s 0,hostbridge -s 1,lpc -s 2,nvme,/dev/zvol/vm/k8s-worker-3 -s 3,ahci-cd,/mnt/vm/isos/ubuntu-24.04-live-server-amd64.iso -s 5,virtio-net,tap9 -s 4,passthru,4/0/0 -s 6,fbuf,tcp=0.0.0.0:6007,w=1280,h=720,vga=io -s 7,xhci,tablet test bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory. GPU passthrough might not work properly. bhyve: gvt_d_setup_opregion: Invalid OpRegion signature bhyve: gvt_d_init: Unable to setup OpRegion Device emulation initialization error: No such file or directory The machine is an AMD EPYC 7343 on a ASRock ROMED8-2T, it has a dedicated onboard GPU (attached to the BMC), which the BIOS is configured to encourage operating systems to use as the primary display - and sure enough the loader and BSD console are presented through the BMCs ASPEED AST2500. Digging through the freebsd source tree, it appears that that pci_gvt-d.c is responsible for this, if it's Intel and a Display then attempt the set up of graphics memory. https://github.com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6cb79075e1830b97594bef3/usr.sbin/bhyve/amd64/pci_gvt-d.c#L44 Has anyone experimented with this? Can I just patch this out, rebuild bhyve and expect a chance of success? Cheers, P. -- *Peter Wood* peter@alastria.net --000000000000562b7a062ae72d63 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Happy new year all.

I= 9;ve been using bhyve happily for a year or two now, multiple machines with= VT-d running happily passing SAS cards and USB cards into VMs.
<= br>
I've reached the point where I want to pass a GPU in for = accelerated encoding/decoding/etc (scrypted, tdarr, jellyfin). I picked up = an Intel ARC A380, as it's encoders/decoders are pretty decent for my u= se case - I'd also seen that there had been success with people using t= he iGPU's in Intel CPUs successfully.

Unfortun= ately after attaching the GPUs PCI device to ppt, and attempting to start a= VM with it attached, I'm greated by an error that seems to focus on iG= PUs?

/usr/sbin/bhyve -A -H -w -u -S -c 2 -m 8G= -l com1,/dev/nmdm202B -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI= .fd,/usr/local/var/cache/bmd/k8s-worker-3.vars -s 0,hostbridge -s 1,lpc -s = 2,nvme,/dev/zvol/vm/k8s-worker-3 -s 3,ahci-cd,/mnt/vm/isos/ubuntu-24.04-liv= e-server-amd64.iso -s 5,virtio-net,tap9 -s 4,passthru,4/0/0 -s 6,fbuf,tcp= =3D0.0.0.0:6007,w=3D1280,h=3D720,vga=3D= io -s 7,xhci,tablet test
bhyve: Warning: Unable to reuse host address of= Graphics Stolen Memory. GPU passthrough might not work properly.
bhyve:= gvt_d_setup_opregion: Invalid OpRegion signature
bhyve: gvt_d_init: Una= ble to setup OpRegion
Device emulation initialization error: No such fil= e or directory

The machine is an AMD EPYC 7343= on a ASRock ROMED8-2T, it has a dedicated onboard GPU (attached to the BMC= ), which the BIOS is configured to encourage operating systems to use as th= e primary display - and sure enough the loader and BSD console are presente= d through the BMCs ASPEED AST2500.

Digging through the freebsd source= tree, it appears that that pci_gvt-d.c is responsible for this, if it'= s Intel and a Display then attempt the set up of graphics memory.
https://github.= com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6cb79075e1830b97594bef3/usr.s= bin/bhyve/amd64/pci_gvt-d.c#L44

Has anyone exp= erimented with this? Can I just patch this out, rebuild bhyve and expect a = chance of success?

Cheers,

P.--
Peter Wood
<= a href=3D"mailto:peter@alastria.net" target=3D"_blank">peter@alastria.net

--000000000000562b7a062ae72d63--