Report of my virtual network lab migrated from virtualbox to bhyve

[Bryan Venteicher]

----- Original Message -----
> Hi Olivier,
> > just a report of my migration to bhyve.
> 
>   Yeah !!!
> 
> > First remark comparing the disk format: With an original nanobsd disk image
> > of 488MB.
> > - Virtualbox format disk size: 133M
> > - bhyve raw disk size: 488M
> 
>   If you create a sparse file for the bhyve raw disk (e.g. with truncate
> -s), du will show the actual blocks used rather than the total size.
> 
> > Second remark regarding the difference between bhyve and virtualbox with
> > disk image management:
> > I'm using the "Linked clone" feature of virtualbox: This avoid to duplicate
> > full guest disk. Here is with the example of 10 VM:
> > - Virtualbox consumed disk space: 133M + 10x133K = about 135M in total
> > - bhyve consumed disk space: 488M x 10 = 4880M in total
> > => Hard drive space is not expensive today, but there is a huge difference
> > (I didn't compare perfs).
> 
>   bhyve doesn't support any VM sparse disk formats - it currently relies
> on the underlying filesystem for this (sparse files, ZFS
> compression/dedup etc). This can perform better than VM sparse formats,
> in both i/o speed and also space usage, at the expense of having to do a
> conversion to/from a flat file if you are importing/exporting.
> 
>   That being said, I'm sure bhyve will gather direct support for these
> at some point.
> 
> > Third remark regarding memory usage on host (reported by top) with each VM
> > configured for 256M of RAM (the guest report 70M of used memory) in
> > SIZE/RES :
> > . Virtualbox consumed RAM: 490/308M x 10 + 125/20M (VBoxSVC) + 80/11M
> > (VBoxXPCOMIPCD) = total of 5100/4010M
> > - bhyve consumed RAM: 283/22M x 10 = total of 2830/220M
> > => Wow... bhyve memory usage is lot's lower than with Virtualbox !
> 
>   To be fair to vbox, the memory usage reported for bhyve is only the
> pages that have been touched by the bhyve process in it's mmap() of
> guest memory. This may be less than that actually in use. We need to
> export the pages in use in the kernel vmspace that represents guest
> memory.
> 
> > Forth remark regarding network card emulation:
> > Virtualbox permits to emulate em(4) NIC that supports altq(4) but bhyve
> > emulate only vtnet(4) that didn't have altq(4) support.
> > => I can't use bhyve for simulating network lab with altq(4)
> 
>   There is an em(4) emulation slowly being worked on. It should also be
> possible to add altq functionality to FreeBSD's virtio net driver.
> 


You have to recompile the driver with VTNET_LEGACY_TX defined to disable
multiqueue support.


> > Fifth remark regarding network card virtualization:
> > Virtualbox permits to generate "internal only" NIC between VM, but bhyve
> > supports only tap. Can you add epair(4) support into bhyve roadmap ?
> > => I have to create a huge number of bridge/tap interfaces on the host just
> > for internal VM Ethernet links. And a simple user can't create bridge/tap
> > interfaces.
> 
>   For internal-only networks, there will most likely be a user-space
> ethernet switch (ala VDE) that bhyve network interfaces can be pointed at.
> 
> > Sixth remark regarding the use of nmdm device for serial redirection:
> > Often, I have to start a connection to the nmdm device for "un-pausing" the
> > starting of the bhyve guest. And once I connect to the nmdm dev, this error
> > is displayed (on head):
> > Assertion failed: (error == 0), function init_pci, file
> > /src/usr.sbin/bhyve/pci_emul.c, line 1047.
> 
>   Yeah, there are some issues with nmdm's simulated modem control :(
> Looking into this.
> 
> > Seventh remark regarding user: I didn't have to use Virtualbox as root
> > (just to be in the virtualbox group), but it's mandatory with bhyve.
> 
>   Fixing this is on the roadmap.
> 
> > And final remark: FreeBSD booting speed is very impressive with bhyve !
> 
>   This may be a useful side-effect of the separate loader process. It
> runs in user-space with Unix process i/o as opposed to a BIOS loader
> which has to run in polled-mode.
> 
> later,
> 
> Peter.
> 
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscribe at freebsd.org"
>