git: 2fc0dfb4ba - main - Status/2024Q1/bhyve.adoc: Add report

From: Lorenzo Salvadore <>
Date: Fri, 29 Mar 2024 16:32:33 UTC
The branch main has been updated by salvadore:


commit 2fc0dfb4ba3b057545fb5643d035552667fc7ee5
Author:     Chris Moerz <>
AuthorDate: 2024-03-29 16:32:11 +0000
Commit:     Lorenzo Salvadore <>
CommitDate: 2024-03-29 16:32:18 +0000

    Status/2024Q1/bhyve.adoc: Add report
    Reviewed by:    status (Pau Amma <>)
 .../en/status/report-2024-01-2024-03/bhyve.adoc    | 72 ++++++++++++++++++++++
 1 file changed, 72 insertions(+)

diff --git a/website/content/en/status/report-2024-01-2024-03/bhyve.adoc b/website/content/en/status/report-2024-01-2024-03/bhyve.adoc
new file mode 100644
index 0000000000..7e94c66526
--- /dev/null
+++ b/website/content/en/status/report-2024-01-2024-03/bhyve.adoc
@@ -0,0 +1,72 @@
+=== Bhyve Improvements
+Links: +
+link:[bhyve production users calls] URL: link:[]
+link:[FreeBSD Wiki - Enterprise Working Group] URL: link:[] +
+link:[FreeBSD Wiki - EWG - bhyve and jails management tooling] URL: link:[] +
+link:[Jan Bramkamp's work on s6rc] URL:[] +
+link:[vmstated on Github] URL: link:[] +
+link:[YouTube - vmstated explained] URL: link:[]
+Contact: Chris Moerz <>
+==== Bhyve I/O Performance Measurements
+Participants of the weekly bhyve production users calls recently discussed bhyve's I/O performance.
+Various ways of measuring and comparing were brought up, however it was quickly clear that there is currently no formal analysis and report on this.
+So, we started this effort in the hopes of better understanding the various impacts of configuration options for a guest on its I/O performance.
+We created a set of shell scripts that harness a FreeBSD guest for running package:benchmarks/fio[] I/O performance measurements under various configurations.
+This allows us to compare multiple criteria like bandwidth, latency, IOPS, and more.
+So far, we are testing for
+* different storage backends (i.e. ahci-hd, nvme, virtio-blk)
+* different memory settings
+* different CPU pinning options
+* different block sizes for the backing storage
+* different block sizes for accessing virtual disks
+We are also pitting results for different CPU manufacturers against each other and contrasting guest vs host performance to better understand the performance impact of virtualization.
+We plan to continue discussing our results during Michael Dexter's weekly bhyve production users call - come join us if you are interested.
+We also hope to be able to present the results at EuroBSDCon in Q3.
+==== Bhyve Virtual Machine Tooling
+Last year, Greg Wallace at the FreeBSD Foundation founded the Enterprise Working Group with the specific goal of addressing pain points of Enterprise users of FreeBSD.
+One of the work groups that emerged clustered around bhyve and jails management tooling.
+After collecting a set of desired features and functionality, one overarching key point for bhyve emerged: the desire to have configuration concepts and tooling for bhyve like the ones available for jails.
+While other desirable features were identified as well, i.e. TPM software emulation and snapshot/restore/host-migration, the conceptual tooling question won over those due to the lower degree of complexity and its clarity on goal and the path on how to take steps towards it.
+Technically, this means working out existing gaps around process supervision and virtual machine state management.
+First steps were taken by experimenting with existing frameworks (i.e. s6rc work by Jan Bramkamp) and eventually -- through discussions in the weekly bhyve production user's calls (organized by Michael Dexter) -- this led to a proof-of-concept implementation of "vmstated".
+Started as an experiment to better understand the problem space of process supervision and virtual machine state handling, vmstated is constructed of a daemon and vmstatedctl management utility.
+It is built with base-only tooling and libraries and leverages FreeBSD specific constructs like kqueue to minimize its resource impact.
+vmstated is configured via a UCL configuration file (similar to [.filename]#jails.conf#) and -- in combination with a man:bhyve_config|5] configuration file -- already provides highest flexibility in configuring virtual machines.
+vmstatedctl provides a jail-like command set to start, stop, and retrieve status information about guests.
+State transitions can easily be hooked via shell scripts and allow running additional commands for network or storage set up and tear down when relevant state changes occur.
+An initial release is already in ports as package:sysutils/vmstated[] and updates are pending commit; however, the newest version can be found on GitHub.
+We are considering expanding the work; we would also like to invite anyone interested to join us in this work!
+Patches, suggestions, feedback, etc. are all very much welcome!
+If you want to know more about our work, come join us at one of Michael Dexter's weekly bhyve production users calls or reach me[by email].
+==== Documentation ===
+We managed to update a few parts of the Handbook and Porter's Handbook (thanks to[Ed Maste],[Joseph Mingrone],[Pau Amma], and[]):
+* several improvements and expansions to the virtualization chapter in the FreeBSD Handbook
+** using a man:bhyve_config|5] configuration file
+** jailing bhyve
+** experimental snapshot and restore feature
+** setting up a Windows guest
+* we also have a review (link:[D43940]) up for an initial step to improving the bhyve man page
+** this was intentionally started with a structural update first to separate the many `-s` flag options
+** once this lands, we can move to a more widespread update to the overall content
+Feedback is obviously very welcome -- on the existing content as well as any additional content we should be looking into!