bhyve -D not cleaning up after itself

From: David E. Cross <>
Date: Sat, 27 Nov 2021 07:40:57 UTC
I have noticed for awhile that bhyve -D doesn't seem to actually do what 
is claimed  (to destroy a VM on guest initiated power-off).  This 
evening I decided to ktrace it to see if I was just not getting 
something about how this was supposed to work, and found:

  68613 vcpu 0   CALL 
  68613 vcpu 0   SCTL "hw.vmm.destroy"
  68613 vcpu 0   RET   __sysctlbyname -1 errno 1 Operation not permitted
  68613 vcpu 0   CALL  exit(0x1)

Reading quickly the kernel code for vm_destroy(), I find 2 candidates:

static int
vmm_priv_check(struct ucred *ucred)

         if (jailed(ucred) &&
             !(ucred->cr_prison->pr_allow & pr_allow_flag))
                 return (EPERM);

         return (0);

This doesn't seem to be it, my process is not jailed.

That leads to the only other (I think) call in sysctl_vmm_destroy that 
could return EPERM:

error = sysctl_handle_string(oidp, buf, buflen, req);

But I am just not seeing it.  Also this EXACT same call works from the 
context of bhyvectl --vm=FOO --destroy, run from the same shell as the 
bhyve process that just terminated.  Is the 'ctx' somehow incorrect in 
bhyve?  I is used earlier in that function, so I am assuming it is right?

Any help appreciated!