Re: Bhyve live migration, virtio-ballooning, kvm-clock

Thank you Matthew for the quick response. I certainly don't want to be a
point of contention for the direction the team is already headed in. Ill
read over what you sent out and if there's anyway I can contribute (lab
hardware, resources, testing), i'd be more than happy to.

Much appreciated!

On Tue, Jun 16, 2026 at 4:27 PM Matthew Grooms <mgrooms@shrew.net> wrote:

>
> On 6/16/26 14:28, William Mckenzie wrote:
>
> Hi all,
>
> For some time we have been working on getting bhyve live vm-migration
> working. We have developed, deployed, and validated three feature series
> against the FreeBSD base system (15.0) and we would like to contribute them
> upstream. I’m writing to ask whether a member of the virtualization team
> would be willing to act as champion/mentor for these series through the
> review process.
>
> What we’ve done:
>
> 1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 commits,
> the
>    kernel engine decomposed into four buildable commits).
>    Live migration of a running guest between two hosts: a versioned
>    VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM precopy driven by
>    EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer reusing
> the
>    existing vm_snapshot machinery, "bhyve -M send/recv" as the userland
>    mover, and a set of restore-correctness fixes (vCPU allocation order,
>    authoritative RIP, PIT re-arm, vm_restore_time on finalize, TSC/vHPET
>    co-anchoring). The PCI BAR re-registration fix is a standalone commit
>    because it also repairs a pre-existing bug in stock bhyvectl(8)
>    --checkpoint/restore, independent of migration. Validated end-to-end on
> a two-host
>    physical Intel lab as a transparent live handoff: a running Rocky Linux
> 9
>    guest migrates in both directions keeping its boot_id, uptime,
> processes,
>    AND live network sessions across the cutover, at ~0.4 s idle downtime;
> 20/20
>    bidirectional runs with zero failures, and a stress run (4 GB / 24 GB
> guest
>    under ~2 GB/s memory churn during the migration) stayed correct with
>    downtime scaling as expected with the at-pause dirty set. One read-only
>    ioctl is added to the capsicum allow-list; all state-changing ioctls
> stay
>    outside the sandbox.
>
> 2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit).
>    A virtio-balloon (type 5) device emulation: inflate/deflate virtqueues
>    with host reclaim via paddr_guest2host() + madvise(MADV_FREE), standard
>    num_pages/actual config space, VIRTIO_BALLOON_F_STATS_VQ guest
> telemetry,
>    and a per-VM control socket created before cap_enter(). Guest-validated
>    against FreeBSD virtio_balloon(4) on two nodes (inflate/deflate tracked
>    exactly; mid-flight readings prove the values are guest-driven) and a
>    Linux guest for the stats queue.
>
> 3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default
>    off). A KVM-compatible paravirtual clock: KVM CPUID signature at
>    0x40000100 (bhyve's own signature leaf untouched),
>    MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on both VMX and SVM
>    paths, publishing standard pvclock structures through
>    vm_gpa_hold_global(). This is the durable fix for Linux guests marking
>    the TSC unstable and degrading to hpet after any snapshot/restore or
>    migration. Validated on hardware: guests register kvm-clock and survive
>    repeated bidirectional migrations with zero TSC-unstable events (the
>    pre-kvmclock baseline reliably degraded on the same hardware).
>
>
> I’ve got a full submission document (design, per-failure bring-up history,
> complete test matrix, untested-areas inventory, and security analysis) and
> the git-format-patch series (against releng/15.0, where they are validated).
>
> I’ve tested many rounds of live vm-migrations across hosts (AMD using KVM
> nested virtualization and Intel physical systems) and have finally gotten
> it to a stable state with 30+ live migrations without packets dropping.  I
> intend to do further testing (specifically with AMD physical boxes).
>
> Bhyve is phenomenal. If there is no interest in a champion, I still intend
> to at least attempt to see the process through (acceptance or not). Happy
> to provide the documentation/requested info.
>
>
> Thanks for working on this. Live migration patch sets have been proposed a
> few times before. You can find the most recent attempt sitting in reviews
> from 2022 ...
>
> https://reviews.freebsd.org/D34722
> https://reviews.freebsd.org/D34811
> https://reviews.freebsd.org/D34719
> https://reviews.freebsd.org/D34720
> https://reviews.freebsd.org/D34721
>
> You should also be able to locate several email threads related to the
> topic on the public freebsd mailing list archives. I won't rehash that
> here, but there was resistance. The orignal work for that and other bhyve
> related projects ( libvdsk w/ qcow2+vmdk support, user mode usb
> pass-through, etc ... ) were hosted here ...
>
> https://github.com/orgs/FreeBSD-UPB/repositories
>
> You should probably also have a look at this ...
>
> https://www.freebsd.org/status/report-2025-10-2025-12/bhyve-cpuid/
>
> From what I gather from his Zagreb presentation, the feature is being
> developed as a foundational layer to import illumos bhyve migration code
> with an eye towards feature parity and potential interoperability.
>
> Good luck!
>
> -Matthew
>