suspend/resume on BHyVe

[Neel Natu]

Hi Lori,

On Mon, Mar 25, 2013 at 11:53 AM, Iori　YONEJI <fivo.11235813 at gmail.com> wrote:
> Hello.
>
> I'm thinking of adding suspend/resume features on BHyVe.
>

Fantastic!

> For this, I think those below must be implemented.
>
>   - virtual machine state command interface
>   - saving registers per CPU
>   - dumping physical memory
>   - saving virt-io and other device emulation state
>
>
> To save registers, the sysctl used in bhyvectl (vmmctl command
> previously) is helpful,
> however, it's interface is no good because getting register value
> cause a sysctl call
> so to context-switch per one register, and for getting all registers,
> it's not efficient.
> I think it's more preferable to make a struct to set or unset boolean
> fields per register,
> to tell which registers kernel should return, and kernel returns those
> state with struct vmxctx.
>
> struct vmxctx is such.
>  struct vmxctx {
>         register_t      tmpstk[32];             /* vmx_return() stack */
>         register_t      tmpstktop;
>
>         register_t      guest_rdi;              /* Guest state */
>         register_t      guest_rsi;
>                 :
>                 :
>
>
> And, considering memory dump, /dev/vmm/vmname is a file that is a map
> of guest memory,
> so memory dump doesn't seem hard, just stop vm, write back all guest
> cache, and copy
> memory file to a regular file.
>
> Finally, I don't know much about device state, but I think there must
> some state to be saved, like
> network stack.
>
> I'm not sure I wrote former, so I appreciate your ideas and suggestions.
>

I think that you are on the right track.

A brute force way of figuring out all the state must be saved is to
look at all the initialization functions that are called when a vm and
a vcpu are created. So, this would be vm_create() and vcpu_init() in
the kernel module.

There is also the hardware assist state that is maintained by the
processor (VT-x or SVM) and this includes things like guest
interruptibility, guest run state etc. I am assuming that it would be
sufficient to save the VMCS page after telling the processor to flush
any state it may be caching on chip.

There is also emulated pci bus, virtio devices and legacy isa device
state that would need to be saved by the userspace 'bhyve' process.

And finally there is the matter of how to communicate with 'bhyve'
process that it needs to suspend the virtual machine and write its
state to disk - perhaps a signal would be good enough place to start.

This certainly sounds like an interesting and challenging project and
we would be happy to help in any way we can.

best
Neel

> Thanks, Iori.
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"