Detect of BHyve VM was powered off or rebooted?

[Aryeh Friedman]

On Sun, Feb 9, 2014 at 5:18 PM, Craig Rodrigues <rodrigc at freebsd.org> wrote:

>
>
>
> On Sun, Feb 9, 2014 at 1:32 PM, Aryeh Friedman <aryeh.friedman at gmail.com>wrote:
>
>>
>>
>>
>> On Sun, Feb 9, 2014 at 4:11 PM, Craig Rodrigues <rodrigc at freebsd.org>wrote:
>>
>>> Hi,
>>>
>>> I posted some rc.d scripts that I am using to boot a BHyve VM
>>> and send the output to a serial console using the /dev/nmdm
>>> driver:
>>>
>>>
>>> http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-January/002040.html
>>>
>>> It works quite well.  There is some things I would like to improve,
>>> and would like some advice on the best way to do it.
>>>
>>> (1)  If the VM was destroyed with bhyvectl --destroy --vm ${VM_NAME},
>>>       then I do not want to automatically restart the VM in the script.
>>>       User should manually:  service bhyvevm start
>>>
>>> (2)  If the VM was powered down, via shutdown -p, or halt -p,
>>>        then in my script I do not want to restart the VM in the script.
>>>        User should manually:  service bhyvevm start
>>>
>>> (3)   If the VM was rebooted via "reboot" or "shutdown -r",
>>>        then I *do* want the script to restart the VM.
>>>
>>> I think if I change my start_vm.sh script to do something like:
>>>
>>>
>>>
>>> (
>>>  while [ -e /dev/vmm/${VM} ]; do
>>>     /usr/sbin/bhyve -c 16 -m 8G -A -H -P -g 0 -s 0:0,hostbridge -s
>>> 1:0,lpc
>>> -s 2:0,virtio-net,${TAP} -s 3:0,virtio-blk,${IMG} -l com1,${CONS_A}
>>> ${VM}"
>>>  done
>>>
>>> )  &
>>>
>>>
>>> then this might cover cases (1) and (3), but what will cover
>>> case (2)?
>>>
>>> Thanks for any advice.
>>>
>>
>> The options you gave (which are really the only ones available) do not
>> distinguish between the reason for the termination of the instance of bhyve
>> pointing to /dev/vmm/XXX  (it just does a normal termination).
>>
>
> Did you play with any of these flags to bhyvectl?
>
>        [--get-vmcs-exit-ctls]
>        [--get-vmcs-exit-reason]
>        [--get-vmcs-exit-qualification]
>        [--get-vmcs-exit-interruption-info]
>
> If I do:
> bhyvectl --get-vmcs-exit-reason --vm vm1
>
> I get:
> vmcs_exit_reason[0]     0x000000000000001e
>
> If I look at:
>
>
> http://svnweb.freebsd.org/base/head/sys/amd64/vmm/intel/vmcs.h?view=markup#l310
>
> we have:
> #define EXIT_REASON_INOUT 30
>

You will likely need to detect the following ones also (all for different
reasons that may or may not have anything to do with an issue with the VM
instead of some host [OS or hardware] issue):

#define EXIT_REASON_EXT_INTR 1
#define EXIT_REASON_INIT 3
#define EXIT_REASON_HLT 12
#define EXIT_REASON_CPUID 10
#define EXIT_REASON_VMLAUNCH 20


> Linux has this stuff also:
> http://lxr.free-electrons.com/source/arch/x86/include/asm/vmx.h?v=3.1#L250
>
> so I guess these values are defined in some Intel manual.
>
>
>
>> In my playing with this for almost a month the only solution seems to be
>> put a trigger on instance shutdown via whatever method to leave a "cookie"
>> file behind on the host (note I have not nor plan to do this in any of my
>> own work).
>>
>
> Yes, I can see why this might need to be done.  I might need to do that
> myself.  However, if bhyvectl can be used to provide adequate status
> information as to why the VM exited, I might be able to avoid this.
>
> It would be nicer though if /usr/sbin/bhyve returned
> the EXIT_REASON in its status code.
>
>
>
>>
>>   An other question is it wise to auto-reboot regardless of reason of the
>> hypervisor termination?
>>
>
> I think the answer is "it depends".  If someone
> does "reboot" or "shutdown -r", I think it is reasonable to restart the
> VM.  If someone does "halt -p" or "shutdown -p", I think it is reasonable
> to not restart the VM.  For any other error condition, I think it is
> reasonable to not restart the VM.
>
> I think that there is no  single answer for all users and all applications.
> In my case, I am running BHyve VM's as "headless" which
> I access via the /dev/nmdm driver.  If the VM reboots normally, I want it
> to restart.
>

It seems almost any boot failure would be host side for example if there is
no SSH access (which if the instance is never logged into then it is likely
a host issue) then it is highly unlikely the VM is in bootable form (this
is after a manual reboot of it)... an other reason (just learned the
hardway) is the user uses a mismatch between backing file formats (my case
using raw images with ahci-hd)
-- 
Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org