From nobody Mon Jan 06 13:56:54 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YRbMk5hzJz5kZy9 for ; Mon, 06 Jan 2025 13:57:06 +0000 (UTC) (envelope-from peter@alastria.net) Received: from mail-qt1-x834.google.com (mail-qt1-x834.google.com [IPv6:2607:f8b0:4864:20::834]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YRbMj6nbPz4gpv for ; Mon, 6 Jan 2025 13:57:05 +0000 (UTC) (envelope-from peter@alastria.net) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x834.google.com with SMTP id d75a77b69052e-4678cd314b6so137877881cf.3 for ; Mon, 06 Jan 2025 05:57:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alastria.net; s=google; t=1736171825; x=1736776625; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=lMpsNfZ814QUAVn5GkrGngUxZs08ZUMjI3LrUuK3GFM=; b=rGJP/a6fARF5PSklzIA+ThY+aJs3BuYlELDh5aNq35lpyvc8k1K718rGiUrz3sEOQH fixx5+0SbyqL/m4Fyj6yuyFUKyIQmAaeClcmcQ3ZMDbyfqRMwTBqP9p+L/yiQiQdsErf IK/eMyKQ8X/6fTqFSFKFWdsOAeakxn89K9wRxQp5DTzesoD9PqPzs3amklExYh6SzJoc C4aBe4NCrhGJUCZNKlGIO0Kq0MzgSQw38sgjsY3Ad2H240UFPUejKg+ES90NW/zY5Tk7 I3psdf1OC/gqOK55kTds9keVtIorjdZR8vFlWTo6fX0eF5sPuWFlg38Ur+mS2BzwQmRs Bgrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736171825; x=1736776625; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lMpsNfZ814QUAVn5GkrGngUxZs08ZUMjI3LrUuK3GFM=; b=PNV+Kwrn5Kb9RPrtLrfXlOm/KrvWxN4uPMCixdcbj2rYSTyhbDvlkY9AlxiYn5/CSa Xu+ZlLcoJsfHGBYarxwtI9+Supkyg4b77cbV8UPzrMj591J7ZfT2SeUZp/dGw36SG7OD pJJWggczJuCRgRPhwtWt6I51HY/W5ftH0X3EEsOBkA9H6CvPzXkdSkMkyZgYDm+zVz8R +wxtMssW2cKQa03qQ0fmc49qzD8/KJPjbv6oKfyy0yd/SxX9ZOq+OudSpSplWOKGvVJj HVqwvRoQFEV9h+uikRn0y2R4oG6pM3v4IdGLrK4uDchAVg74MJ8r+OmB2oGQQtKqZDRl E3Rg== X-Gm-Message-State: AOJu0Yx7k1sPEymWjdGpYGWGhtNHrhvckGzi5y2g1UAGs7DT+LwaMYB+ J6X4b5G3VJtyvaZ2kcWvu6dR5vdVmRNiN66ipFV/DrqaqJEp6FpMFlBIQWP6I50uHweaq3PyC42 PXsUo5Tm8o1CDnZVExJnF6LMf6m8LEe4h/l3zwczOBHC35/e3hsY= X-Gm-Gg: ASbGncs4khpK/hY4AQc8KcmE49HhEzImQn4pHtUBEhHl9nF8BQCJl0VI9PM83eXI/fT B/7ZuZ3/0G/gf5Yqu1L4YEs7bD+9qnUYNN+NVHuqJxdVY9ANqdlIDilQlMrOOBFB5spBPdw== X-Google-Smtp-Source: AGHT+IEkPJp5LN47HcH/oVCy36rZ30TjAcRIGSWwl8BzP1nFpGnIPUevwN4ctQgaTSRLaIPQCyICn+hHTzcVoAbSjcU= X-Received: by 2002:a05:622a:24c:b0:466:a6e4:4c1a with SMTP id d75a77b69052e-46a4a8e6c07mr772387351cf.31.1736171825214; Mon, 06 Jan 2025 05:57:05 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Peter Wood Date: Mon, 6 Jan 2025 13:56:54 +0000 Message-ID: Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? To: =?UTF-8?Q?Corvin_K=C3=B6hne?= Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000e946f0062b0a0059" X-Rspamd-Queue-Id: 4YRbMj6nbPz4gpv X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --000000000000e946f0062b0a0059 Content-Type: text/plain; charset="UTF-8" Thanks for the feedback Corvin, and thank you for the hard work you've been putting into GPU passthru. I'm reaching the end of my limited knowledge here, and I have no expectation of any further assistance - but as a status update, BIOS in CSM (with legacy video op rom, so I see the console). If you want to pass the option rom to the guest, you can use the rom option > of > passthru devices: > > -s 1/2/3,passthru,1/2/3,rom=/path/to/rom I extracted the option ROM using linux, I was able to use the /sys/devices/pci*/rom route to extract it, it seems valid at a glance (768k dump) - but no idea how to really tell. Using the patched bhyve executable to bypass gvt-d: -s 4/0/0,passthru,4/0/0,rom=/mnt/vm/intel-arc-a380.bin -s 5/0/0,passthru,5/0/0 (5/0/0 is a separate audio device exposing the audio channels of the HDMI ports). Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux 6.8.0) VM still fails: [ 2.508656] i915 0000:00:04.0: enabling device (0000 -> 0002) [ 2.520226] i915 0000:00:04.0: [drm] Local memory IO size: 0x000000017c800000 [ 2.520232] i915 0000:00:04.0: [drm] Local memory available: 0x000000017c800000 [ 2.540148] i915 0000:00:04.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none [ 2.550829] i915 0000:00:04.0: [drm] Finished loading DMC firmware i915/dg2_dmc_ver2_08.bin (v2.8) [ 2.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc size changed from 32768 to 36864 [ 2.565855] i915 0000:00:04.0: [drm] GT0: GuC firmware i915/dg2_guc_70.bin version 70.20.0 [ 2.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware i915/dg2_huc_gsc.bin version 7.10.3 [ 2.565979] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc size changed from 32768 to 36864 [ 2.567001] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status = 0x40000056, time = 0ms, freq = 2300MHz, ret = 0 [ 2.567006] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status: Reset = 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01 [ 2.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware production part check failure [ 2.567077] i915 0000:00:04.0: [drm] *ERROR* GT0: GuC initialization failed -ENOEXEC [ 2.567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enabling uc failed (-5) [ 2.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed to initialize GPU, declaring it wedged! [ 2.570106] i915 0000:00:04.0: [drm:add_taint_for_CI [i915]] CI tainted:0x9 by intel_gt_set_wedged_on_init+0x34/0x50 [i915] [ 2.587048] [drm] Initialized i915 1.6.0 20230929 for 0000:00:04.0 on minor 1 Interestingly intel_gpu_top will interact with the card to a degree, it shows a render utilization (of 0%), but none of the other card capabilities. There are some very similar errors in Google which may suggest it's may not be a bhyve/passthru issue, though it could be. I need to spin up a new VM with more bleeding edge linux (or maybe even Win11) to see if it can talk to the card. https://github.com/intel-analytics/ipex-llm/issues/12122 I'll post if I get any further, but I suspect this is the end for now. Peter. -- *Peter Wood* peter@alastria.net --000000000000e946f0062b0a0059 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks for the feedback Corvin, and = thank you for the hard work you've been putting into GPU passthru.

I'm reaching the end of my limited knowledge here,= and I have no expectation of any further assistance - but as a status upda= te, BIOS in CSM (with legacy video op rom, so I see the console).
=

If you wan= t to pass the option rom to the guest, you can use the rom option of
passthru devices:

-s 1/2/3,passthru,1/2/3,rom=3D/path/to/rom

= I extracted the option ROM using linux, I was able to use the /sys/devices/= pci*/rom route to extract it, it seems valid at a glance (768k dump) - but = no idea how to really tell.

Using the patched = bhyve executable to bypass gvt-d: -s 4/0/0,passthru,4/0/0,rom=3D/mnt/vm/int= el-arc-a380.bin -s 5/0/0,passthru,5/0/0
(5/0/0 is a separate audi= o device exposing the audio channels of the HDMI ports).

= Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux 6.8.0) V= M still fails:
[ =C2=A0 =C2=A02.508656] i915 0000:00:04.0: enabli= ng device (0000 -> 0002)
[ =C2=A0 =C2=A02.520226] i915 0000:00:04.0: = [drm] Local memory IO size: 0x000000017c800000
[ =C2=A0 =C2=A02.520232] = i915 0000:00:04.0: [drm] Local memory available: 0x000000017c800000
[ = =C2=A0 =C2=A02.540148] i915 0000:00:04.0: vgaarb: VGA decodes changed: oldd= ecodes=3Dio+mem,decodes=3Dnone:owns=3Dnone
[ =C2=A0 =C2=A02.550829] i915= 0000:00:04.0: [drm] Finished loading DMC firmware i915/dg2_dmc_ver2_08.bin= (v2.8)
[ =C2=A0 =C2=A02.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS = capture alloc size changed from 32768 to 36864
[ =C2=A0 =C2=A02.565855] = i915 0000:00:04.0: [drm] GT0: GuC firmware i915/dg2_guc_70.bin version 70.2= 0.0
[ =C2=A0 =C2=A02.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware = i915/dg2_huc_gsc.bin version 7.10.3
[ =C2=A0 =C2=A02.565979] i915 0000:0= 0:04.0: [drm] GT0: GUC: ADS capture alloc size changed from 32768 to 36864<= br>[ =C2=A0 =C2=A02.567001] i915 0000:00:04.0: [drm] GT0: GUC: load failed:= status =3D 0x40000056, time =3D 0ms, freq =3D 2300MHz, ret =3D 0
[ =C2= =A0 =C2=A02.567006] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status:= Reset =3D 0, BootROM =3D 0x2B, UKernel =3D 0x00, MIA =3D 0x00, Auth =3D 0x= 01
[ =C2=A0 =C2=A02.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware = production part check failure
[ =C2=A0 =C2=A02.567077] i915 0000:00:04.0= : [drm] *ERROR* GT0: GuC initialization failed -ENOEXEC
[ =C2=A0 =C2=A02= .567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enabling uc failed (-5)
[= =C2=A0 =C2=A02.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed to ini= tialize GPU, declaring it wedged!
[ =C2=A0 =C2=A02.570106] i915 0000:00:= 04.0: [drm:add_taint_for_CI [i915]] CI tainted:0x9 by intel_gt_set_wedged_o= n_init+0x34/0x50 [i915]
[ =C2=A0 =C2=A02.587048] [drm] Initialized i915 = 1.6.0 20230929 for 0000:00:04.0 on minor 1

Interes= tingly intel_gpu_top will interact with the card to a degree, it shows a re= nder utilization (of 0%), but none of the other card capabilities. There ar= e some very similar errors in Google which may suggest it's may not be = a bhyve/passthru issue, though it could be. I need to spin up a new VM with= more bleeding edge linux (or maybe even Win11) to see if it can talk to th= e card.


I'll post if I get any further, b= ut I suspect this is the end for now.

Peter.
--
<= /div>
--000000000000e946f0062b0a0059--