Understanding Bhyve shutdown

[Matt Churchyard]

As I understand it

1) shutdown from guest
2) 'kill <pid>' -> pressing the power button once.
3) --force-poweroff -> holding power button in
4) --force-reset -> pressing the reset button
5) bhyvectl --destroy -> same as 3? (although vmm is destroyed as well)

1 or 2 are obviously preferred. I've found however that some guests don't respond that well to the apci event. CentOS 6 seems to need apci installing and enabling?!, and my Win2012R2 machine will only shutdown if I send the signal twice.

The rest are all hard shutdowns that should not be seen as a way to stop the guest, just a last resort if it can't be stopped correctly. I don't know the practical difference between poweroff&destroy vs just destroy.

I see no reason why the documentation suggests reboot rather than shutdown. Bhyve exits either way and the only difference is the exit code. When using one of the bhyve management tools that support reboots (such as vmrun.sh/vm-bhyve/iohyve) however, a "restart" exit code (0) will cause the guest to restart automatically; If a guest is locked up for example, a --force-reset will cause it to immediately reboot.

>From a host, the only safe choice is 'kill <pid>' (possibly multiple times) and wait for bhyve process to (hopefully) exit. If that doesn't work either the acpi issue needs fixing or ideally the guest needs to be stopped using its built-in shutdown function.

The devs will have to confirm why they're implemented the way they are. My instinct is that bhyvectl works on the vmm device, and reset/poweroff are just functions that are possible via that interface, and so have been exposed. The ACPI shutdown event likely needs to be initiated by the bhyve process itself, hence using a signal to it. /end-speculation

I think most users will want to use a bhyve tool so the raw specifics of the bhyve/bhyvectl commands are glossed over, although that doesn't mean the handbook documentation of the base commands shouldn't be as complete/correct as possible of course.

Matt

-----Original Message-----
From: owner-freebsd-virtualization at freebsd.org [mailto:owner-freebsd-virtualization at freebsd.org] On Behalf Of Roman Bogorodskiy
Sent: 13 April 2016 11:55
To: freebsd-virtualization at FreeBSD.org
Subject: Understanding Bhyve shutdown

Hi,

I was trying to get better understanding of how to properly shutdown VMs in bhyve, but unfortunately the documentation does not provide much details on that.

Specifically, handbook [I] suggests to reboot a machine and then run bhyvectl --destroy on it. 

The bhyvectl(8) manpage mentions the '--force-reset' and '--force-poweroff' switches, but does not give details on those. 

I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html

I tried all the options I know and wrote down the results. I also have some questions, hopefully you'll be able to answer some of them.

1. bhyvectl --vm=$name --destroy

 * looks like hard poweroff in the guest
 * the corresponding bhyve(8) process goes away
 * /dev/vmm/ entry goes away

In my experience, it's a dangerous way to shutdown a VM because sometimes it appears it damages the image and VM fails to boot with something like this:

---
Starting devd.
mode = 0100600, inum = 170269, fs = /
panic: ffs_valloc: dup alloc
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80984e30 at kdb_backtrace+0x60
#1 0xffffffff809489e6 at vpanic+0x126
#2 0xffffffff809488b3 at panic+0x43
#3 0xffffffff80b74a6e at ffs_valloc+0x84e
#4 0xffffffff80bb60ad at ufs_makeinode+0x7d
#5 0xffffffff80bb24fd at ufs_create+0x2d
#6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1
#7 0xffffffff809cd9e6 at uipc_bindat+0x346
#8 0xffffffff809c5488 at kern_bindat+0x108
#9 0xffffffff809c52a7 at sys_bind+0x77
#10 0xffffffff80d4b3f7 at amd64_syscall+0x357
#11 0xffffffff80d30adb at Xfast_syscall+0xfb
Uptime: 3s

Dump failed. Partition too small.
---

2. kill -SIGTERM $bhyve_pid

If guest supports ACPI shutdown:

 * guest shuts down cleanly
 * the corresponding bhyve(8) process terminates
 * /dev/vmm entry is still here, need bhyvectl --destroy for complete
   cleanup

If guest does not support ACPI shutdown (such as doing sysctl
hw.acpi.power_button_state=NONE):

 * Nothing happens

 Q1: Is there a way to know if a guest reacted to power button but
     waiting for the bhyve process to terminate?   
 Q2: Why it's not done via bhyvectl (it seems that it's easier for users
     + don't have to overload a useful SIGTERM signal)

3. bhyvectl --vm=$name --force-poweroff

 * looks like hard poweroff in the guest
 * the corresponding bhyve(8) process goes away
 * /dev/vmm entry is still here, need bhyvectl --destroy for complete
   cleanup

Q: what's the practical difference with just doing --destroy right away?

4. bhyvectl --vm=$name --force-reset

Looks very similar to item #3 with just different exit code (reboot
appears to be using 0, while shutdown and halt use 1 and 2).   

Q: what's the practical use of it?

Would greatly appreciate if somebody could provide more details on that.
I guess we'll need to update Handbook with this information as well because it needs to mention SIGTERM for ACPI shutdown at least. 

Roman Bogorodskiy
_______________________________________________
freebsd-virtualization at freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"