Re: ERRATA CORRIGE "BUG !!!! X64 Exception Type - 06(#UD - Invalid Opcode) CPU Apic ID - 00000000 !!!! while trying to pass thru my devices with bhyve on FreeBSD 14-CURRENT"

From: Aryeh Friedman <aryeh.friedman_at_gmail.com>
Date: Thu, 10 Feb 2022 14:39:46 UTC
Deleted all the screen dumps since my question is more general:

What happens if you do not turn on Xorg on the host and start the VM from
the console (likely should make no difference but worth a try).

In your original post you never mentioned it worked on 14-CURRENT but not
on 13-RELEASE (w/ patches).   If this is the case it is almost certainly a
bug, but the general policy of most developers is not to make bug fixes for
older versions.   This means you are likely stuck with 14-CURRENT.

On Thu, Feb 10, 2022 at 9:24 AM Mario Marietto <marietto2008@gmail.com>
wrote:

> Hello.
>
> I've bought an additional PCI to USB (Renesas) controller that I've
> attached to the PCIe Gen3 x4 (Type 22110) of my MOBO Gigabyte Aorus Pro
> with Intel I9 chipset and Nvidia Geforce RTX 2080 ti. On this controller
> I've attached two USB disks and my KInect 2. The nVidia geforce RTX 2080 ti
> is attached to the PCIe Gen3 x 4 (Type 2280). Check this image :
> https://ibb.co/fC0jR1p
>
> check by yourself the logs showing that my passed through devices are
> working great :
>
> dmesg --> https://pastebin.ubuntu.com/p/cSy7jkHdZz/
> pciconf -vl --> https://pastebin.ubuntu.com/p/xZNggrz6D5/
>
> On FreeBSD 14 the bhyve setup below works. I can boot Linux passing
> through the devices that you see. But I have used a raw image file (not a
> physical installation on a sata or nvme disk,because in this case it wont
> work and for this I've opened another bug). With this I want to demonstrate
> that there aren't any problems with my bios settings and with my passed
> thru devices.
>
> bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H \
> -s 0,hostbridge \
> -s 1,virtio-blk,/mnt/da1p2/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \
> -s 3:0,passthru,2/0/0 \
> -s 3:1,passthru,2/0/1 \
> -s 3:2,passthru,2/0/2 \
> -s 3:3,passthru,2/0/3 \
> -s 4:0,passthru,1/0/0 \
> -s 9,virtio-net,tap0 \
> -s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
> -s 30,xhci,tablet \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
> -l com1,stdio \
> vm0
>
> My host graphic card is not an Nvidia card. It's the graphic chipset
> integrated on the motherboard (CoffeeLake-S GT2 [UHD Graphics 630); to use
> it I've installed the xf86-video-intel driver with this xorg.conf file :
>
> Section "Device"
> Identifier  "Card0"
> Driver      "intel"
> BusID       "PCI:0:2:0"
> EndSection
>
> so,now,let's to straight to this problem :
>
> I've created a new VM with the goal to passthru my graphic card and my usb
> controller from the host os (freebsd 14-CURRENT) to the guest os (freebsd
> 13R p7) and I've used the parameters below :
>
> bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
> -s 0,hostbridge \
> -s 1,ahci-hd,/mnt/da2p2/bhyve/os/BSD/freebsd13vm.img \
> -s 2,passthru,1/0/0 \
> -s 3:0,passthru,2/0/0 \
> -s 3:1,passthru,2/0/1 \
> -s 3:2,passthru,2/0/2 \
> -s 3:3,passthru,2/0/3 \
> -s 8,virtio-net,tap1 \
> -s 9,virtio-9p,sharename=/ \
> -s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \
> -s 30,xhci,tablet \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
> -l com1,stdio \
> vm1
>
> inside /etc/rc.conf I've added the following parameters :
>
> linux_enable="YES"
> linux_mounts_enable="YES"
> vm_enable="YES"
> kdm5_enable="YES"
> dbus_enable="YES"
> sddm_enable="YES"
>
> on /boot/loader.conf :
>
> nvidia_load="YES"
> vmm_load="YES"
> nmdm_load="YES"
> if_tap_load="YES"
> if_bridge_load="YES"
> bridgestp_load="YES"
> linprocfs_load="YES"
> linsysfs_load="YES"
> tmpfs_load="YES"
> verbose_loading="YES"
> hw.vmm.topology.cores_per_package=8
> kern.racct.enable=1
> snd_hda_load="YES"
>
> on /etc/sysctl.conf
>
> net.link.tap.up_on_open=1
> net.inet.ip.forwarding=1
> net.inet.ip.random_id=1
> kern.evdev.rcpt_mask=6
> kern.init_shutdown_timeout="900"
> kern.capmode_coredump=1
>
> and with this xorg.conf file :
>
> Section "ServerLayout"
>        Identifier     "X.org Configured"
>        Screen      0  "Screen0" 0 0
>        #Screen      1  "Screen1" RightOf "Screen0"
>        InputDevice    "Mouse0" "CorePointer"
>        InputDevice    "Keyboard0" "CoreKeyboard"
> EndSection
>
> Section "Module"
>        Load  "glx"
> EndSection
>
> Section "Files"
>        ModulePath   "/usr/local/lib/xorg/modules"
>        FontPath     "/usr/local/share/fonts/misc/"
>        FontPath     "/usr/local/share/fonts/TTF/"
>        FontPath     "/usr/local/share/fonts/OTF/"
>        FontPath     "/usr/local/share/fonts/Type1/"
>        FontPath     "/usr/local/share/fonts/100dpi/"
>        FontPath     "/usr/local/share/fonts/75dpi/"
>        FontPath     "catalogue:/usr/local/etc/X11/fontpath.d"
> EndSection
>
> Section "InputDevice"
>        Identifier  "Mouse0"
>        Driver      "mouse"
>        Option      "Protocol" "auto"
>        Option      "Device" "/dev/sysmouse"
>        Option      "ZAxisMapping" "4 5 6 7"
> EndSection
>
> Section "Monitor"
>        Identifier   "Monitor0"
>        VendorName   "Monitor Vendor"
>        ModelName    "Monitor Model"
> EndSection
>
> Section "Device"
>        ### Available Driver options are:-
>        ### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
>        ### <string>: "String", <freq>: "<f> Hz/kHz/MHz",
>        ### <percent>: "<f>%"
>        ### [arg]: arg optional
>        #Option     "SWcursor"                  # [<bool>]
>        #Option     "kmsdev"                    # <str>
>        #Option     "ShadowFB"                  # [<bool>]
>        #Option     "AccelMethod"               # <str>
>        #Option     "PageFlip"                  # [<bool>]
>        #Option     "ZaphodHeads"               # <str>
>        #Option     "DoubleShadow"              # [<bool>]
>        #Option     "Atomic"                    # [<bool>]
>        Identifier  "Card0"
>        Driver      "nvidia"
>        BusID       "PCI:0:4:0"
> EndSection
>
> this is what happens when I boot it :
>
> Loading kernel...
> /boot/kernel/kernel text=0x17b9e0 text=0xdd6d50 text=0x65ba0c data=0x140
> data=0x1b9348+0x445cb8 syms=[0x8+0x178ed8+0x8+0x1990a2]
> Loading configured modules...
> snd_hda...if_bridge.../boot/kernel/if_bridge.ko size 0xfc48 at 0x2112000
> loading required module 'bridgestp'
> /boot/kernel/bridgestp.ko size 0x8178 at 0x2122000
> tmpfs.../boot/kernel/tmpfs.ko size 0x14820 at 0x212b000
> /etc/hostid.../etc/hostid size=0x25
> /boot/entropy.../boot/entropy size=0x1000
> if_tap...linprocfs.../boot/kernel/linprocfs.ko size 0x10e50 at 0x2141000
> loading required module 'linux_common'
> /boot/kernel/linux_common.ko size 0x2c650 at 0x2152000
> nvidia.../boot/modules/nvidia.ko size 0x28c8640 at 0x217f000
> loading required module 'linux'
> /boot/kernel/linux.ko size 0x9e070 at 0x4a48000
> nmdm.../boot/kernel/nmdm.ko size 0x4110 at 0x4ae7000
> bridgestp...linsysfs.../boot/kernel/linsysfs.ko size 0x79b8 at 0x4aec000
> vmm.../boot/kernel/vmm.ko size 0x5748f0 at 0x4af4000
> Start @ 0xffffffff8037c000 ...
> efi_check_space: Unable to expand staging area
> efi_check_space: Unable to expand staging area
> EFI framebuffer information:
> addr, size     0xc3000000, 0x1000000
> dimensions     1440 x 900
> stride         1440
> masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
> !!!! X64 Exception Type - 06(#UD - Invalid Opcode)  CPU Apic ID - 00000000
> !!!!
> RIP  - 00000000000C000E, CS  - 0000000000000038, RFLAGS - 0000000000010086
> RAX  - 0000000054A218D8, RCX - 0000000000000002, RDX - 0000000000000002
> RBX  - 0000000005069001, RSP - 00000000BFF58098, RBP - 00000000BFF58100
> RSI  - 0000000000001000, RDI - 0000000040001000
> R8   - 0000000000000001, R9  - 00000000BFF580D8, R10 - 00000000B902D6E0
> R11  - 0000000000000D80, R12 - 00000000BE819050, R13 - 0000000005069000
> R14  - 00000000BE819058, R15 - 00000000BE819048
> DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
> GS   - 0000000000000030, SS  - 0000000000000030
> CR0  - 0000000080010033, CR2 - 0000000000000000, CR3 - 00000000BFC01000
> CR4  - 0000000000000668, CR8 - 0000000000000000
> DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
> DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
> GDTR - 00000000BFBEEA98 0000000000000047, LDTR - 0000000000000000
> IDTR - 00000000BF832018 0000000000000FFF,   TR - 0000000000000000
> FXSAVE_STATE - 00000000BFF57CF0
> !!!! Can't find image information. !!!!
>
> it freezes here and I can't do anything anymore.
>
> --
> Mario.
>


-- 
Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org