KVM working on Talos2/POWER9!

Tue Jul 10 23:14:24 UTC 2018

Hello,

FreeBSD is running in KVM on the Talos II/POWER9 with some caveats.

Before doing anything, you need to append "disable_radix=yes" to the 
Linux kernel command line and reboot.

I am able to install it by manually running:
qemu-system-ppc64 -enable-kvm -m 8192 -nographic -vga none -mem-prealloc 
-mem-path /dev/hugepages -cdrom 
FreeBSD-12.0-CURRENT-powerpc-powerpc64-20180709-r336134-disc1.iso -drive 
file=freebsd-ppc.qcow2,format=qcow2 -smp 2

Not the most pressing issue, but during the install, I learned the hard 
way that the GPT partition table format doesn't work. I was able to get 
it running with MBR (which is the default option anyway). I haven't 
tried any of the other options. With GPT, it fails with the same 
messages as found here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211599

I am then able to run the VM manually with the following:
qemu-system-ppc64 -enable-kvm -m 8192 -nographic -vga none -mem-prealloc 
-mem-path /dev/hugepages -drive file=freebsd-ppc.qcow2,format=qcow2 -smp 2

I haven't tried earlier snapshots of 12.0-CURRENT or 11.2 since figuring 
out how to get it running yet.

When attempting to use KVM via libvirt, FreeBSD begins to boot, and then 
is unable to find the filesystem to mount root. This happens when 
booting off of the install disc, as well as off of the hard disk image 
itself even after I successfully installed running kvm manually. Here I 
am booting off of the virtual hard disk:
http://vasteel.neo-zeon.de/~hiryu/freebsd-power9.png

Here is how libvirt tries to run this VM:
/usr/bin/qemu-system-ppc64 -name guest=freebsd-ppc,debug-threads=on -S 
-object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-31-freebsd-ppc/master-key.aes 
-machine pseries-2.12,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 
4096 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/31-freebsd-ppc 
-realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 
ffd4b906-7af7-48d4-8275-f990c97faaa4 -no-user-config -nodefaults 
-chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-31-freebsd-ppc/monitor.sock,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc 
-no-shutdown -boot strict=on -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 
-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device 
ahci,id=sata0,bus=pci.0,addr=0x7 -drive 
file=/var/lib/libvirt/images/freebsd-ppc.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-drive if=none,id=drive-scsi0-0-0-1,readonly=on -device 
scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1 
-netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:34:14,bus=pci.0,addr=0x1 
-chardev pty,id=charserial0 -device 
spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -device 
usb-kbd,id=input0,bus=usb.0,port=1 -device 
usb-mouse,id=input1,bus=usb.0,port=2 -vnc 127.0.0.1:0 -device 
VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x6 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny 
-msg timestamp=on

I see a backtrace in dmesg that doesn't always seem occur. Pastebin 
appears to be currently down for me, so here's the full dmesg in case 
other details are helpful:

Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r336134: Mon Jul  9 18:14:23 UTC 2018
root at releng3.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 
powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
cpu0: IBM POWER9 revision 2.2, 2234.00 MHz
cpu0: Features 
dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE>
cpu0: Features2 
eee00000<ARCH207,HTM,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN>
real memory  = 8544436224 (8148 MB)
avail memory = 8249282560 (7867 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
random: unblocking device.
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree> on nexus0
xicp0: <External Interrupt Presentation Controller> on ofwbus0
xicp0: Handling CPUs 0-15
vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0
vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781322 on vdevice0
vscsi0: Queue depth 22 commands
llan0: <POWER Hypervisor Virtual Ethernet> irq 16781321 on vdevice0
llan0: Ethernet address: 00:12:34:56:00:00
uart0: <POWER Hypervisor Virtual Serial Port> irq 16781315 on vdevice0
pcib0: <RTAS Host-PCI bridge> on ofwbus0
pci0: <POWER Hypervisor PCI bus> on pcib0
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
cpu1: <Open Firmware CPU> on cpulist0
rtas0: <Run-Time Abstraction Services> on ofwbus0
rtas0: registered as a time-of-day clock, resolution 0.002000s
Timecounter "timebase" frequency 512000000 Hz quality 0
Event timer "decrementer" frequency 512000000 Hz quality 1000
Timecounters tick every 1.000 msec
llan0: link state changed to UP
usb_needs_explore_all: no devclass
cd0 at vscsi0 bus 0 scbus0 target 0 lun 8200
cd0: <QEMU QEMU CD-ROM 2.5+> Removable CD-ROM SPC-3 SCSI device
cd0: 150.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
da0 at vscsi0 bus 0 scbus0 target 0 lun 0
da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
da0: 150.000MB/s transfers
da0: 32768MB (67108864 512 byte sectors)
SMP: AP CPU #1 launched
Trying to mount root from ufs:/dev/da0s2a [rw]...
WARNING: WITNESS option enabled, expect reduced performance.
lock order reversal:
  1st 0xe000000000c8a3b8 bufwait (bufwait) @ 
/usr/src/sys/kern/vfs_bio.c:3916
  2nd 0xc000000007521600 dirhash (dirhash) @ 
/usr/src/sys/ufs/ufs/ufs_dirhash.c:289
stack backtrace:
#0 0xc0000000006c937c at .witness_debugger+0x90
#1 0xc0000000006cc650 at .witness_checkorder+0xa20
#2 0xc00000000064f870 at ._sx_xlock+0xbc
#3 0xc000000000989f74 at .ufsdirhash_acquire+0x3c
#4 0xc00000000098a6e4 at .ufsdirhash_add+0x30
#5 0xc00000000098e250 at .ufs_direnter+0x7e8
#6 0xc0000000009975ac at .ufs_rename+0xb5c
#7 0xc000000000a49a88 at .VOP_RENAME_APV+0x164
#8 0xc0000000007553fc at .kern_renameat+0x3c0
#9 0xc0000000007555f4 at .sys_rename+0x2c
#10 0xc000000000a2db70 at .trap+0x65c
#11 0xc000000000a22824 at .powerpc_interrupt+0x29c
lo0: link state changed to UP

During shutdown, I saw another backtrace, this one seems to occur all 
the time or at least more frequently:
All buffers synced.
lock order reversal:
  1st 0xc00000000747c9c8 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1343
  2nd 0xc00000000747d248 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2572
stack backtrace:
#0 0xc0000000006c937c at .witness_debugger+0x90
#1 0xc0000000006cc650 at .witness_checkorder+0xa20
#2 0xc0000000006179c8 at .lockmgr_lock_fast_path+0x1a0
#3 0xc00000000073407c at .vop_stdlock+0x30
#4 0xc000000000a4850c at .VOP_LOCK1_APV+0x158
#5 0xc00000000075cd98 at ._vn_lock+0xac
#6 0xc00000000074d15c at .vget+0xac
#7 0xc0000000004dfafc at .devfs_allocv+0x110
#8 0xc0000000004dcec0 at .devfs_root+0x54
#9 0xc00000000074e4c4 at .vflush+0x5e8
#10 0xc0000000004dcfd4 at .devfs_unmount+0x64
#11 0xc00000000073ea3c at .dounmount+0xaec
#12 0xc000000000746570 at .unmount_or_warn+0x28
#13 0xc000000000746668 at .vfs_unmountall+0x88
#14 0xc000000000723b28 at .bufshutdown+0x308
#15 0xc000000000642ffc at .kern_reboot+0x2f0
#16 0xc000000000643de0 at .sys_reboot+0x454
#17 0xc000000000a2db70 at .trap+0x65c

Hope this helps!

-Cameron