Multiple bugs on bhyve present on FreeBSD 13R p7 while passing thru USB controller and graphic card

From: Mario Marietto <marietto2008_at_gmail.com>
Date: Tue, 08 Feb 2022 16:20:00 UTC
Hello to everyone.

I'm using bhyve on FreeBSD 13R p7 and I'm trying to pass thru my usb
controller and / or my graphic card while at the same time I'm trying to
boot 3 operating systems (Ubuntu,Windows and FreeBSD 13) installed on a
SATA and / or NVME disk using the virtio-blk driver and the ahci-hd driver.
The problems that come out from this setup are different and concatenated.
Anyway,those OS won't boot because it seems that the devices that I pass
thru interferes with the booting process. If I don't pass thru any
device,they can boot properly. In addition,my graphic card (geforce 2080
ti) seems to have an additional problem. It seems that it is not passed
correctly.

CASE 0) Without passing thru the usb controller and the graphic card ubuntu
or windows 11 can boot properly with these parameters :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

BdsDxe: loading Boot0001 "UEFI BHYVE SATA DISK BHYVE-51F3-061F-A385" from
PciRoot(0x0)/Pci(0x1,0x0)/Sata(0x0,0
xFFFF,0x0)
BdsDxe: starting Boot0001 "UEFI BHYVE SATA DISK BHYVE-51F3-061F-A385" from
PciRoot(0x0)/Pci(0x1,0x0)/Sata(0x0,
0xFFFF,0x0)

CASE 1) Passing thru the usb controller and the graphic card and booting
from a physical installation of Linux or Windows 11 on a nvme disk :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,passthru,1/0/0 \
-s 3:0,passthru,2/0/0 \
-s 3:1,passthru,2/0/1 \
-s 3:2,passthru,2/0/2 \
-s 3:3,passthru,2/0/3 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

Sometimes Assertion failed: (error == 0), function modify_bar_registration,
file /usr/src/usr.sbin/bhyve/pci_emul.c, line 501: Abort / sometimes black
screen frozen

CASE 2) Passing thru the usb controller and booting from a physical
installation of Linux or Windows 11 on a nvme disk :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,passthru,1/0/0 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

= BdsDxe: failed to load Boot0001 "UEFI BHYVE SATA DISK
BHYVE-51F3-061F-A385" from PciRoot(0x0)/Pci(0x1,0x0)/Sat
a(0x0,0xFFFF,0x0): Not Found

CASE 3) Passing thru the graphic card and booting from a physical
installation of Linux or Windows 11 on a nvme disk :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2:0,passthru,2/0/0 \
-s 2:1,passthru,2/0/1 \
-s 2:2,passthru,2/0/2 \
-s 2:3,passthru,2/0/3 \
-s 8,virtio-net,tap1 \
-s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
vm1

= Assertion failed: (error == 0), function modify_bar_registration, file
/usr/src/usr.sbin/bhyve/pci_emul.c, line 501 : Abort

I've tried with virtio-blk instead of ahci-hd and with /dev/ada2 (sata
disk) instead of the nvdm disk and nothing is changed. now,explain to me
how should I make the bug report. I'm not sure,since I see two different or
more bugs concatenated but I haven't the competence to report them
separately. Would be better if you do it for me and then give me the links
of the bugs,so I learn something more.

-- 
Mario.