Bhyve live migration, virtio-ballooning, kvm-clock

Hi all,

For some time we have been working on getting bhyve live vm-migration working. We have developed, deployed, and validated three feature series against the FreeBSD base system (15.0) and we would like to contribute them upstream. I’m writing to ask whether a member of the virtualization team would be willing to act as champion/mentor for these series through the review process.

What we’ve done:

1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 commits, the
   kernel engine decomposed into four buildable commits).
   Live migration of a running guest between two hosts: a versioned
   VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM precopy driven by
   EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer reusing the
   existing vm_snapshot machinery, "bhyve -M send/recv" as the userland
   mover, and a set of restore-correctness fixes (vCPU allocation order,
   authoritative RIP, PIT re-arm, vm_restore_time on finalize, TSC/vHPET
   co-anchoring). The PCI BAR re-registration fix is a standalone commit
   because it also repairs a pre-existing bug in stock bhyvectl(8)
   --checkpoint/restore, independent of migration. Validated end-to-end on a two-host
   physical Intel lab as a transparent live handoff: a running Rocky Linux 9
   guest migrates in both directions keeping its boot_id, uptime, processes,
   AND live network sessions across the cutover, at ~0.4 s idle downtime; 20/20
   bidirectional runs with zero failures, and a stress run (4 GB / 24 GB guest
   under ~2 GB/s memory churn during the migration) stayed correct with
   downtime scaling as expected with the at-pause dirty set. One read-only
   ioctl is added to the capsicum allow-list; all state-changing ioctls stay
   outside the sandbox.

2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit).
   A virtio-balloon (type 5) device emulation: inflate/deflate virtqueues
   with host reclaim via paddr_guest2host() + madvise(MADV_FREE), standard
   num_pages/actual config space, VIRTIO_BALLOON_F_STATS_VQ guest telemetry,
   and a per-VM control socket created before cap_enter(). Guest-validated
   against FreeBSD virtio_balloon(4) on two nodes (inflate/deflate tracked
   exactly; mid-flight readings prove the values are guest-driven) and a
   Linux guest for the stats queue.

3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default
   off). A KVM-compatible paravirtual clock: KVM CPUID signature at
   0x40000100 (bhyve's own signature leaf untouched),
   MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on both VMX and SVM
   paths, publishing standard pvclock structures through
   vm_gpa_hold_global(). This is the durable fix for Linux guests marking
   the TSC unstable and degrading to hpet after any snapshot/restore or
   migration. Validated on hardware: guests register kvm-clock and survive
   repeated bidirectional migrations with zero TSC-unstable events (the
   pre-kvmclock baseline reliably degraded on the same hardware).


I’ve got a full submission document (design, per-failure bring-up history, complete test matrix, untested-areas inventory, and security analysis) and the git-format-patch series (against releng/15.0, where they are validated).

I’ve tested many rounds of live vm-migrations across hosts (AMD using KVM nested virtualization and Intel physical systems) and have finally gotten it to a stable state with 30+ live migrations without packets dropping.  I intend to do further testing (specifically with AMD physical boxes).

Bhyve is phenomenal. If there is no interest in a champion, I still intend to at least attempt to see the process through (acceptance or not). Happy to provide the documentation/requested info.

Thanks for the consideration.

William Mckenzie
wmckenzie@rhelitpro.com