Re: Bhyve live migration, virtio-ballooning, kvm-clock

On 6/16/26 14:28, William Mckenzie wrote:
> Hi all,
>
> For some time we have been working on getting bhyve live vm-migration 
> working. We have developed, deployed, and validated three feature 
> series against the FreeBSD base system (15.0) and we would like to 
> contribute them upstream. I’m writing to ask whether a member of the 
> virtualization team would be willing to act as champion/mentor for 
> these series through the review process.
>
> What we’ve done:
>
> 1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 
> commits, the
>    kernel engine decomposed into four buildable commits).
>    Live migration of a running guest between two hosts: a versioned
>    VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM precopy driven by
>    EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer 
> reusing the
>    existing vm_snapshot machinery, "bhyve -M send/recv" as the userland
>    mover, and a set of restore-correctness fixes (vCPU allocation order,
>    authoritative RIP, PIT re-arm, vm_restore_time on finalize, TSC/vHPET
>    co-anchoring). The PCI BAR re-registration fix is a standalone commit
>    because it also repairs a pre-existing bug in stock bhyvectl(8)
>    --checkpoint/restore, independent of migration. Validated 
> end-to-end on a two-host
>    physical Intel lab as a transparent live handoff: a running Rocky 
> Linux 9
>    guest migrates in both directions keeping its boot_id, uptime, 
> processes,
>    AND live network sessions across the cutover, at ~0.4 s idle 
> downtime; 20/20
>    bidirectional runs with zero failures, and a stress run (4 GB / 24 
> GB guest
>    under ~2 GB/s memory churn during the migration) stayed correct with
>    downtime scaling as expected with the at-pause dirty set. One read-only
>    ioctl is added to the capsicum allow-list; all state-changing 
> ioctls stay
>    outside the sandbox.
>
> 2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit).
>    A virtio-balloon (type 5) device emulation: inflate/deflate virtqueues
>    with host reclaim via paddr_guest2host() + madvise(MADV_FREE), standard
>    num_pages/actual config space, VIRTIO_BALLOON_F_STATS_VQ guest 
> telemetry,
>    and a per-VM control socket created before cap_enter(). Guest-validated
>    against FreeBSD virtio_balloon(4) on two nodes (inflate/deflate tracked
>    exactly; mid-flight readings prove the values are guest-driven) and a
>    Linux guest for the stats queue.
>
> 3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default
>    off). A KVM-compatible paravirtual clock: KVM CPUID signature at
>    0x40000100 (bhyve's own signature leaf untouched),
>    MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on both VMX and SVM
>    paths, publishing standard pvclock structures through
>    vm_gpa_hold_global(). This is the durable fix for Linux guests marking
>    the TSC unstable and degrading to hpet after any snapshot/restore or
>    migration. Validated on hardware: guests register kvm-clock and survive
>    repeated bidirectional migrations with zero TSC-unstable events (the
>    pre-kvmclock baseline reliably degraded on the same hardware).
>
>
> I’ve got a full submission document (design, per-failure bring-up 
> history, complete test matrix, untested-areas inventory, and security 
> analysis) and the git-format-patch series (against releng/15.0, where 
> they are validated).
>
> I’ve tested many rounds of live vm-migrations across hosts (AMD using 
> KVM nested virtualization and Intel physical systems) and have finally 
> gotten it to a stable state with 30+ live migrations without packets 
> dropping.  I intend to do further testing (specifically with AMD 
> physical boxes).
>
> Bhyve is phenomenal. If there is no interest in a champion, I still 
> intend to at least attempt to see the process through (acceptance or 
> not). Happy to provide the documentation/requested info.
>

Thanks for working on this. Live migration patch sets have been proposed 
a few times before. You can find the most recent attempt sitting in 
reviews from 2022 ...

https://reviews.freebsd.org/D34722
https://reviews.freebsd.org/D34811
https://reviews.freebsd.org/D34719
https://reviews.freebsd.org/D34720
https://reviews.freebsd.org/D34721

You should also be able to locate several email threads related to the 
topic on the public freebsd mailing list archives. I won't rehash that 
here, but there was resistance. The orignal work for that and other 
bhyve related projects ( libvdsk w/ qcow2+vmdk support, user mode usb 
pass-through, etc ... ) were hosted here ...

https://github.com/orgs/FreeBSD-UPB/repositories

You should probably also have a look at this ...

https://www.freebsd.org/status/report-2025-10-2025-12/bhyve-cpuid/

 From what I gather from his Zagreb presentation, the feature is being 
developed as a foundational layer to import illumos bhyve migration code 
with an eye towards feature parity and potential interoperability.

Good luck!

-Matthew