Report of my virtual network lab migrated from virtualbox to bhyve

[Olivier Cochard-Labbé]

Hi,

just a report of my migration to bhyve.

I was using VirtualBox for generate full-meshed network lab of multiple VM
(essentially nanobsd based) and have migrated my script to bhyve.

My original script is resumed to this kind of usage:
./lab-script.sh -i FreeBSD-image-disk -n number-of-vm -l number-of-LAN

For example I can easly generate this kind of topology:
./BSDRP-lab-bhyve.sh -i nanobsd-serial.img -n 4 -l 2
VM 1 have the following NIC:
- vtnet0 connected to VM 2.
- vtnet1 connected to VM 3.
- vtnet2 connected to VM 4.
- vtnet3 connected to LAN number 1
- vtnet4 connected to LAN number 2
VM 2 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 3.
- vtnet2 connected to VM 4.
- vtnet3 connected to LAN number 1
- vtnet4 connected to LAN number 2
VM 3 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 4.
- vtnet3 connected to LAN number 1
- vtnet4 connected to LAN number 2
VM 4 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 3.
- vtnet3 connected to LAN number 1
- vtnet4 connected to LAN number 2

First remark comparing the disk format: With an original nanobsd disk image
of 488MB.
- Virtualbox format disk size: 133M
- bhyve raw disk size: 488M

Second remark regarding the difference between bhyve and virtualbox with
disk image management:
I'm using the "Linked clone" feature of virtualbox: This avoid to duplicate
full guest disk. Here is with the example of 10 VM:
- Virtualbox consumed disk space: 133M + 10x133K = about 135M in total
- bhyve consumed disk space: 488M x 10 = 4880M in total
=> Hard drive space is not expensive today, but there is a huge difference
(I didn't compare perfs).

Third remark regarding memory usage on host (reported by top) with each VM
configured for 256M of RAM (the guest report 70M of used memory) in
SIZE/RES :
. Virtualbox consumed RAM: 490/308M x 10 + 125/20M (VBoxSVC) + 80/11M
(VBoxXPCOMIPCD) = total of 5100/4010M
- bhyve consumed RAM: 283/22M x 10 = total of 2830/220M
=> Wow... bhyve memory usage is lot's lower than with Virtualbox !

Forth remark regarding network card emulation:
Virtualbox permits to emulate em(4) NIC that supports altq(4) but bhyve
emulate only vtnet(4) that didn't have altq(4) support.
=> I can't use bhyve for simulating network lab with altq(4)

Fifth remark regarding network card virtualization:
Virtualbox permits to generate "internal only" NIC between VM, but bhyve
supports only tap. Can you add epair(4) support into bhyve roadmap ?
=> I have to create a huge number of bridge/tap interfaces on the host just
for internal VM Ethernet links. And a simple user can't create bridge/tap
interfaces.

Sixth remark regarding the use of nmdm device for serial redirection:
Often, I have to start a connection to the nmdm device for "un-pausing" the
starting of the bhyve guest. And once I connect to the nmdm dev, this error
is displayed (on head):
Assertion failed: (error == 0), function init_pci, file
/src/usr.sbin/bhyve/pci_emul.c, line 1047.

Seventh remark regarding user: I didn't have to use Virtualbox as root
(just to be in the virtualbox group), but it's mandatory with bhyve.

And final remark: FreeBSD booting speed is very impressive with bhyve !

My bhyve and virtualbox scripts (and qemu) used are online here:
http://bsdrp.net/documentation/examples/how_to_build_a_bsdrp_router_lab

Regards,

Olivier