bhyve(8) suspend interface

From: Roman Bogorodskiy <novel_at_freebsd.org>
Date: Sat, 29 Nov 2025 14:20:05 UTC
Hi,

While experimenting with the bhyve suspend/resume feature I noticed a
couple of things which seem a little confusing.

The first thing is bhyve(8) exit code after suspend. If I run:

  bhyve -c 2 -m 4096 ... my_vm

And then run:

  bhyvectl --suspend=/var/run/vms/my_vm --vm my_vm

Once suspend completes, the bhyve process exits with retcode 0.
According to the bhyve(8) manual page, this exit code corresponds to a
VM reboot. It also seems inconvenient from management software/scripting
point of view, because reaction to this exit code is to boot a VM again
(assuming that if a user initiated a reboot, they expect their VM to
boot again soon). My impression is that it would be more natural to
group it with other events like 1:power off or 2:halted, which do not
imply immediate VM restart. Or add a dedicated exit code, say, 5:suspended?

The second question is that, apparently, calling bhyvectl --suspend ...
is not enough, as after suspend completion the VM entity is still
present in /dev/vmm, so it looks like bhyvectl --destroy is necessary.

It's not obvious from the bhyvectl(8) manual page and generally it's
not obvious why two bhyvectl(8) commands are necessary to suspend a VM.
Interestingly, bhyvectl(8) does not complain early about using
"--suspend" and "--destroy" at the same time, i.e.:

  bhyvectl --suspend=/var/run/vms/my_vm --destroy --vm my_vm

But fails with obscure messages:

  bhyvectl(8): connect() failed: Connection refused
  bhyve(8):

  vm_run error -1, errno 6
  Failed to emulate instruction sequence [ 8b800040000089c048018108060000 ] at 0xffffffffc06d85f3

And dumps a core.

Roman