Understanding Bhyve shutdown

Neel Natu neelnatu at gmail.com
Tue Apr 19 18:02:29 UTC 2016

Hi Roman,

On Wed, Apr 13, 2016 at 3:55 AM, Roman Bogorodskiy
<bogorodskiy at gmail.com> wrote:
> Hi,
> I was trying to get better understanding of how to properly shutdown VMs
> in bhyve, but unfortunately the documentation does not provide much
> details on that.
> Specifically, handbook [I] suggests to reboot a machine and then run
> bhyvectl --destroy on it.
> The bhyvectl(8) manpage mentions the '--force-reset' and
> '--force-poweroff' switches, but does not give details on those.
> I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
> I tried all the options I know and wrote down the results. I also have
> some questions, hopefully you'll be able to answer some of them.
> 1. bhyvectl --vm=$name --destroy
>  * looks like hard poweroff in the guest
>  * the corresponding bhyve(8) process goes away
>  * /dev/vmm/ entry goes away
> In my experience, it's a dangerous way to shutdown a VM because
> sometimes it appears it damages the image and VM fails to boot with
> something like this:
> ---
> Starting devd.
> mode = 0100600, inum = 170269, fs = /
> panic: ffs_valloc: dup alloc
> cpuid = 0
> KDB: stack backtrace:
> #0 0xffffffff80984e30 at kdb_backtrace+0x60
> #1 0xffffffff809489e6 at vpanic+0x126
> #2 0xffffffff809488b3 at panic+0x43
> #3 0xffffffff80b74a6e at ffs_valloc+0x84e
> #4 0xffffffff80bb60ad at ufs_makeinode+0x7d
> #5 0xffffffff80bb24fd at ufs_create+0x2d
> #6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1
> #7 0xffffffff809cd9e6 at uipc_bindat+0x346
> #8 0xffffffff809c5488 at kern_bindat+0x108
> #9 0xffffffff809c52a7 at sys_bind+0x77
> #10 0xffffffff80d4b3f7 at amd64_syscall+0x357
> #11 0xffffffff80d30adb at Xfast_syscall+0xfb
> Uptime: 3s
> Dump failed. Partition too small.
> ---

Yup, this is biggest hammer you could use to shutdown a virtual
machine. As you noticed, this is usually a bad thing because it does
not give the guest OS an opportunity to shutdown cleanly.

> 2. kill -SIGTERM $bhyve_pid
> If guest supports ACPI shutdown:
>  * guest shuts down cleanly
>  * the corresponding bhyve(8) process terminates
>  * /dev/vmm entry is still here, need bhyvectl --destroy for complete
>    cleanup
> If guest does not support ACPI shutdown (such as doing sysctl
> hw.acpi.power_button_state=NONE):
>  * Nothing happens
>  Q1: Is there a way to know if a guest reacted to power button but
>      waiting for the bhyve process to terminate?

Not really, except to wait for some amount of time to give the guest a
chance to shutdown.

>  Q2: Why it's not done via bhyvectl (it seems that it's easier for users
>      + don't have to overload a useful SIGTERM signal)

It seems natural to overload SIGTERM to do this. For e.g. when the
host is shutting down it will send a SIGTERM to all running processes
and translating this into an ACPI poweroff event for the guest allows
it to shutdown cleanly.

> 3. bhyvectl --vm=$name --force-poweroff
>  * looks like hard poweroff in the guest
>  * the corresponding bhyve(8) process goes away
>  * /dev/vmm entry is still here, need bhyvectl --destroy for complete
>    cleanup
> Q: what's the practical difference with just doing --destroy right away?

'force-poweroff' is equivalent to a hard power off on real hardware.

The only practical difference between '--force-poweroff' and
'--destroy' is that destroy will also remove the device node in
/dev/vmm and release any memory allocated for the guest back to the

> 4. bhyvectl --vm=$name --force-reset
> Looks very similar to item #3 with just different exit code (reboot
> appears to be using 0, while shutdown and halt use 1 and 2).
> Q: what's the practical use of it?

The exit code can be used by scripts on top of 'bhyve' to decide
whether to restart execution of the guest (reset) or stop completely

> Would greatly appreciate if somebody could provide more details on that.
> I guess we'll need to update Handbook with this information as well
> because it needs to mention SIGTERM for ACPI shutdown at least.

Hope that helps.


> Roman Bogorodskiy
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"

More information about the freebsd-virtualization mailing list