Understanding Bhyve shutdown

[Roman Bogorodskiy]

Hi,

I was trying to get better understanding of how to properly shutdown VMs
in bhyve, but unfortunately the documentation does not provide much
details on that.

Specifically, handbook [I] suggests to reboot a machine and then run
bhyvectl --destroy on it. 

The bhyvectl(8) manpage mentions the '--force-reset' and
'--force-poweroff' switches, but does not give details on those. 

I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html

I tried all the options I know and wrote down the results. I also have
some questions, hopefully you'll be able to answer some of them.

1. bhyvectl --vm=$name --destroy

 * looks like hard poweroff in the guest
 * the corresponding bhyve(8) process goes away
 * /dev/vmm/ entry goes away

In my experience, it's a dangerous way to shutdown a VM because
sometimes it appears it damages the image and VM fails to boot with
something like this:

---
Starting devd.
mode = 0100600, inum = 170269, fs = /
panic: ffs_valloc: dup alloc
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80984e30 at kdb_backtrace+0x60
#1 0xffffffff809489e6 at vpanic+0x126
#2 0xffffffff809488b3 at panic+0x43
#3 0xffffffff80b74a6e at ffs_valloc+0x84e
#4 0xffffffff80bb60ad at ufs_makeinode+0x7d
#5 0xffffffff80bb24fd at ufs_create+0x2d
#6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1
#7 0xffffffff809cd9e6 at uipc_bindat+0x346
#8 0xffffffff809c5488 at kern_bindat+0x108
#9 0xffffffff809c52a7 at sys_bind+0x77
#10 0xffffffff80d4b3f7 at amd64_syscall+0x357
#11 0xffffffff80d30adb at Xfast_syscall+0xfb
Uptime: 3s

Dump failed. Partition too small.
---

2. kill -SIGTERM $bhyve_pid

If guest supports ACPI shutdown:

 * guest shuts down cleanly
 * the corresponding bhyve(8) process terminates
 * /dev/vmm entry is still here, need bhyvectl --destroy for complete
   cleanup

If guest does not support ACPI shutdown (such as doing sysctl
hw.acpi.power_button_state=NONE):

 * Nothing happens

 Q1: Is there a way to know if a guest reacted to power button but
     waiting for the bhyve process to terminate?   
 Q2: Why it's not done via bhyvectl (it seems that it's easier for users
     + don't have to overload a useful SIGTERM signal)

3. bhyvectl --vm=$name --force-poweroff

 * looks like hard poweroff in the guest
 * the corresponding bhyve(8) process goes away
 * /dev/vmm entry is still here, need bhyvectl --destroy for complete
   cleanup

Q: what's the practical difference with just doing --destroy right away?

4. bhyvectl --vm=$name --force-reset

Looks very similar to item #3 with just different exit code (reboot
appears to be using 0, while shutdown and halt use 1 and 2).   

Q: what's the practical use of it?

Would greatly appreciate if somebody could provide more details on that.
I guess we'll need to update Handbook with this information as well
because it needs to mention SIGTERM for ACPI shutdown at least. 

Roman Bogorodskiy