bhyve -D not cleaning up after itself

From: David E. Cross <david_at_crossfamilyweb.com>
Date: Sat, 27 Nov 2021 07:40:57 UTC
I have noticed for awhile that bhyve -D doesn't seem to actually do what 
is claimed  (to destroy a VM on guest initiated power-off).  This 
evening I decided to ktrace it to see if I was just not getting 
something about how this was supposed to work, and found:


  68613 vcpu 0   CALL 
__sysctlbyname(0x1ebcdb20a133,0xe,0,0,0x1ebce4ba60f0,0x9)
  68613 vcpu 0   SCTL "hw.vmm.destroy"
  68613 vcpu 0   RET   __sysctlbyname -1 errno 1 Operation not permitted
  68613 vcpu 0   CALL  exit(0x1)


Reading quickly the kernel code for vm_destroy(), I find 2 candidates:

static int
vmm_priv_check(struct ucred *ucred)
{

         if (jailed(ucred) &&
             !(ucred->cr_prison->pr_allow & pr_allow_flag))
                 return (EPERM);

         return (0);
}

This doesn't seem to be it, my process is not jailed.

That leads to the only other (I think) call in sysctl_vmm_destroy that 
could return EPERM:

error = sysctl_handle_string(oidp, buf, buflen, req);


But I am just not seeing it.  Also this EXACT same call works from the 
context of bhyvectl --vm=FOO --destroy, run from the same shell as the 
bhyve process that just terminated.  Is the 'ctx' somehow incorrect in 
bhyve?  I is used earlier in that function, so I am assuming it is right?


Any help appreciated!