Re: how to run bhyve and virtualbox at the same time

The content of the VBox.log strongly smells like bhyve wants to use VT-x/SVM exclusively, unfortunately nothing VirtualBox can do anything about.
vmm.ko/bhyve would need to implement a mode for sharing those capabilities.

And because I forgot to add this to my last mail. I had a look at the vmm.ko API already and found that the guest memory management API looks quite strange and
is completely different from what kvm/AppleHV/Hyper-V etc. do. With those you allocate page aligned memory using mmap()/VirtualAlloc() and pass the GPA
and the userspace virtual pointer to the hypervisor for the mapping. With vmm.ko you allocate the memory through the hypervisor directly using an ioctl
providing the GPA and size and you get a user space mapping by a sub-sequent mmap() call on the character device for the given offset. And with libvmm
you only have the option to allocate all the memory up-front.

Regards,
Alexander Eichner

> On 28.06.2022 15:32, Mario Marietto <marietto2008@gmail.com> wrote:
> 
> I've rebooted and then I have launched virtualbox first and when macos has been loaded totally,I gave the following command :
> 
> sudo kldload vmm
> 
> and this is what happened :
> 
> <Screenshot_2022-06-28_15-09-49.jpg>
> I've attached the log files produced....
> 
> Il giorno mar 28 giu 2022 alle ore 14:48 Mario Marietto <marietto2008@gmail.com <mailto:marietto2008@gmail.com>> ha scritto:
> Ok. As root I have disabled the hwvirtexclusive property giving this command :
> 
> VBoxManage setproperty hwvirtexclusive off
> 
> as explained here : https://www.virtualbox.org/manual/ch09.html <https://www.virtualbox.org/manual/ch09.html>
> 
> this is what happens when I launch bhyve first and then virtualbox....
> 
> 
> <Screenshot_2022-06-28_14-12-36.jpg>
> 
> More interesting is to see what happens when I launch virtualbox first,executing this script :
> 
> setxkbmap it
> bhyvectl --vm=vm0 --force-reset
> bhyvectl --vm=vm0 --destroy
> bhyvectl --vm=vm1 --force-reset
> bhyvectl --vm=vm1 --destroy
> bhyvectl --vm=vm2 --force-reset
> bhyvectl --vm=vm2 --destroy
> bhyvectl --vm=vm3 --force-reset
> bhyvectl --vm=vm3 --destroy
> bhyvectl --vm=vm4 --force-reset
> bhyvectl --vm=vm4 --destroy
> bhyvectl --vm=vm5 --force-reset
> bhyvectl --vm=vm5 --destroy
> bhyvectl --vm=vm6 --force-reset
> bhyvectl --vm=vm6 --destroy
> bhyvectl --vm=vm7 --force-reset
> bhyvectl --vm=vm7 --destroy
> bhyvectl --vm=vm8 --force-reset
> bhyvectl --vm=vm8 --destroy
> bhyvectl --vm=vm9 --force-reset
> bhyvectl --vm=vm9 --destroy
> bhyvectl --vm=vm10 --force-reset
> bhyvectl --vm=vm10 --destroy
> bhyvectl --vm=vm11 --force-reset
> bhyvectl --vm=vm11 --destroy
> bhyvectl --vm=vm12 --force-reset
> bhyvectl --vm=vm12 --destroy
> bhyvectl --vm=vm13 --force-reset
> bhyvectl --vm=vm13 --destroy
> bhyvectl --vm=vm14 --force-reset
> bhyvectl --vm=vm14 --destroy
> bhyvectl --vm=vm15 --force-reset
> bhyvectl --vm=vm15 --destroy
> bhyvectl --vm=vm16 --force-reset
> bhyvectl --vm=vm16 --destroy
> bhyvectl --vm=vm17 --force-reset
> bhyvectl --vm=vm17 --destroy
> bhyvectl --vm=vm18 --force-reset
> bhyvectl --vm=vm18 --destroy
> bhyvectl --vm=vm19 --force-reset
> bhyvectl --vm=vm19 --destroy
> bhyvectl --vm=vm20 --force-reset
> bhyvectl --vm=vm20 --destroy
> sudo kldunload vmm
> virtualbox
> 
> At this point MacOS is able to boot and run correctly. At this point I tried to launch a bhyve vm executing this script :
> 
> bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
> -s 0,hostbridge \
> -s 1,nvme,/dev/nvd0,bootindex=1 \
> -s 2,ahci-hd,/dev/$vmdisk2 \
> -s 3,ahci-hd,/dev/$vmdisk3 \
> -s 4,ahci-hd,/dev/$vmdisk4 \
> -s 7,virtio-net,tap4 \
> -s 10,hda,play=/dev/dsp,rec=/dev/dsp \
> -s 29,fbuf,tcp=0.0.0.0:5904 <http://0.0.0.0:5904/>,w=1500,h=950 \
> -s 30,xhci,tablet \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
> vm4 < /dev/null & sleep 2 && vncviewer 0:4
> 
> my FreeBSD 13.1-RELEASE totally crashed. I've seen a lot of errors on the screen,a lot of them related to efi and my PC rebooted. It seems that the file bhyve.core has not produced,even if I read "creating memory dumping"...
> 
> Il giorno mar 28 giu 2022 alle ore 12:03 Alexander Eichner <freebsd@aeichner.de <mailto:freebsd@aeichner.de>> ha scritto:
> Hi,
> 
> VirtualBox 6.1.x uses its own hypervisor with kernel modules on macOS, it just uses some internal kernel interfaces to be able to share
> the hardware virtualization functionality with other hypervisors.
> 
> On hosts other than Windows and macOS VirtualBox defaults to an exclusive use of VT-x/SVM for performance reasons. That mode can be changed
> to do the initialization every time VirtualBox is scheduled on a particular CPU which has higher overhead but allows sharing the hardware virtualization capabilities
> with other hypervisors. The setting can be changed with VBoxManage. Look at the manual under [1] and search for „hwvirtexclusive“.
> However bhyve would need to support something similar to be able to play along, I don’t know whether it does.
> 
> Regards,
> Alexander Eichner
> 
> [1] https://www.virtualbox.org/manual/UserManual.html <https://www.virtualbox.org/manual/UserManual.html>
> 
> > On 28.06.2022 11:29, David Chisnall <theraven@FreeBSD.org> wrote:
> > 
> > On 27/06/2022 20:41, Mario Marietto wrote:
> >> I did it already and it works. Fact is that if I use virtualbox I can't use bhyve at the same time. So It's not a good idea to use only virtualbox machines.
> > 
> > On macOS, VirtualBox can coexist with other hypervisors because it does not provide a kernel module (it is not allowed to on recent macOS) but is, instead, layered on top of Apple's Hypervisor framework.
> > 
> > bhyve has also been ported to sit on top of Apple's Hypervisor framework (xhyve, used by Docker) and I believe that the abstractions provided by the bhyve kernel module are fairly similar.
> > 
> > It should be possible to port VirtualBox to sit on top of vmm.ko.  I have not been able to find any documentation of the vmm(4) ioctls other than the source code, so I'd imagine that about 2/3 of this work would be documenting the kernel interfaces.
> > 
> > David
> > 
> 
> 
> 
> 
> -- 
> Mario.
> 
> 
> -- 
> Mario.
> <logs.zip>