bhyve and virtualbox on the same host using EPT

[Peter Grehan]

Hi Rob,

> Bhyve generates the following error after the bootloader screen:
>
> -----------------------------------------------------
> Booting...
> 4 vCPUs requested but only 1 available
> -----------------------------------------------------

  What this means is that the model of CPU that you have doesn't support 
the "unrestricted guest" feature, which allows a guest to run in 16-bit 
mode.

  bhyve boots FreeBSD directly into 64-bit mode on the main CPU, but 
secondary CPUs are started through a boostrap from 16-bit mode to 64-bit 
mode. There was a hack early on in bhyve to avoid this by starting the 
secondaries directly in 64-bit mode, but it was bhyve- and 
FreeBSD-specific and difficult to maintain. Given that there was only a 
small timeframe of CPUs that supported EPT but not unrestricted guest 
(early Nehalems), we decided to the 64-bit AP spinup.

  You can still run single vCPU guests though.

> The Virtualbox logs below show EPT in use:

  On the issue of running VBox and bhyve at the same time: that's not 
recommended and will probably result in a crash. To do this requires 
cooperation from hypervisors in that they will completely flush state 
any time they could be context-switched away to avoid the other 
hypervisor's state from colliding. This results in poor performance 
which is why it usually isn't done by default.

  I think we know how to fix this in bhyve, but aren't there yet.

later,

Peter.