Linux guest kernel GPF in bhyve host on AMD Ryzen

[Aoife Finch]

Linux guests with recent kernels panic with a GPF during boot on my system:

Ryzen 7 1700 or Ryzen 7 2700
Asus PRIME B350 PLUS - bios 5007
64GB DDR4 RAM
FreeBSD 12.0-p7

Here are boot messages from the Ubuntu live server iso (ubuntu-18.04.2-live-server-amd64.iso):

[    0.024000] Spectre V2 : Spectre mitigation: LFENCE not serializing, switching to generic retpoline
[    0.948696] ACPI Error: Could not enable RealTimeClock event (20170831/evxfevnt-218)
[ TIME ] Timed out waiting for device dev-di…fd\x2db21e\x2dc3186f34105d.device.
[DEPEND] Dependency failed for /subiquity_config.
[  OK  ] Started Uncomplicated firewall.
[  OK  ] Started Create list of required sta…vice nodes for the current kernel.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Started Create Static Device Nodes in /dev.
         Starting udev Kernel Device Manager...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started udev Coldplug all Devices.
[  OK  ] Started Load Kernel Modules.
         Mounting Kernel Configuration File System...
         Starting Apply Kernel Variables...
         Mounting FUSE Control File System...
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Mounted Kernel Configuration File System.
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started LVM2 metadata daemon.
[  OK  ] Started Monitoring of LVM2 mirrors,…sing dmeventd or progress polling.
[  OK  ] Started Set the console keyboard layout.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Local File Systems (Pre).
         Mounting /media/region.lower...
         Mounting /tmp...
         Mounting /media/rack.lower...
         Mounting /media/filesystem...
[  OK  ] Mounted /tmp.
[  OK  ] Reached target Local File Systems.
         Starting ebtables ruleset management...
         Starting Set console font and keymap...
         Starting Create Volatile Files and Directories...
         Starting Tell Plymouth To Write Out Runtime Data...
[  OK  ] Mounted /media/filesystem.
[  OK  ] Mounted /media/region.lower.
[  OK  ] Mounted /media/rack.lower.
         Mounting /media/region...
         Mounting /media/rack...
[  OK  ] Mounted /media/region.
[  OK  ] Mounted /media/rack.
[  OK  ] Started Tell Plymouth To Write Out Runtime Data.
[  OK  ] Started Set console font and keymap.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
         Starting Network Time Synchronization...
[    7.012798] general protection fault: 0000 [#1] SMP NOPTI
[    7.013343] Modules linked in: sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 overlay nls_utf8 isofs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear aesni_intel aes_x86_64 crypto_simd cryptd glue_helper psmouse virtio_net virtio_blk ahci libahci
[    7.016815] CPU: 0 PID: 31 Comm: kworker/0:1 Not tainted 4.15.0-45-generic #48-Ubuntu
[    7.017567] Hardware name:  BHYVE, BIOS 1.00 03/14/2014
[    7.018076] RIP: 0010:switch_mm_irqs_off+0x327/0x400
[    7.018561] RSP: 0018:ffffa957801e7e20 EFLAGS: 00010046
[    7.019070] RAX: 0000000000000001 RBX: ffff971ac91a4a40 RCX: 0000000000000049
[    7.019765] RDX: 0000000000000000 RSI: ffff971ac91a4a40 RDI: ffff971ac91a18c0
[    7.020457] RBP: ffffa957801e7e48 R08: 0000000000000383 R09: 0000000000000000
[    7.021141] R10: 0000000000000000 R11: 0000000000000383 R12: ffff971ac91a18c0
[    7.021828] R13: 0000000000000000 R14: ffff971ac91a4a40 R15: ffff971ac91a18c0
[    7.022520] FS:  0000000000000000(0000) GS:ffff971adfc00000(0000) knlGS:0000000000000000
[    7.023295] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    7.023854] CR2: 00000000000000b0 CR3: 000000001a14c000 CR4: 00000000000406f0
[    7.024547] Call Trace:
[    7.024798]  __schedule+0x252/0x8a0
[    7.025141]  ? pwq_dec_nr_in_flight+0x49/0xa0
[    7.025568]  schedule+0x2c/0x80
[    7.025880]  worker_thread+0xba/0x410
[    7.026232]  kthread+0x121/0x140
[    7.026544]  ? process_one_work+0x410/0x410
[    7.026951]  ? kthread_create_worker_on_cpu+0x70/0x70
[    7.027438]  ret_from_fork+0x35/0x40
[    7.027794] Code: c1 49 8b 06 49 8b 7e 08 49 83 c6 18 31 d2 31 f6 e8 df 73 b8 00 49 8b 06 48 85 c0 75 e7 eb a3 b9 49 00 00 00 b8 01 00 00 00 31 d2 <0f> 30 e9 3d fd ff ff 9c 58 0f 1f 44 00 00 49 89 c5 fa 66 0f 1f 
[    7.029568] RIP: switch_mm_irqs_off+0x327/0x400 RSP: ffffa957801e7e20
[    7.030171] ---[ end trace 1379b5776b2d7813 ]---

Similar crashes occur with other Linux distros running recent kernels. Older kernels (like 4.4.0-135-generic in Ubuntu 16.04) boot and run fine. Current FreeBSD guests run fine.

I swapped the Ryzen 1700 for a 2700 with no change. Unfortunately, I don't have a spare motherboard or ram to swap.

Is this a Ryzen issue or is there something I can tweak to fix this? Thanks!

-- Aoife