bhyve and virtualbox on the same host using EPT

Tue Aug 6 23:10:41 UTC 2013

> On Tue, Aug 6, 2013 at 3:02 PM, <mailinglists at debank.tv> wrote:
> 
>> All,
>> 
>> I'm trying to run a bhyve VM on a host that's already running Virtualbox, the problem I encounter is bhyve doesn't detect EPT capabilities but the Virtualbox logs indicate EPT is in use and functional.
>> My question is; is it possible for two pieces of virtualization software to use EPT on the same host?
>> 
>> See below for the various logs, happy to provide more info if needed.
>> 
>> Bhyve generates the following error after the bootloader screen:
>> 
>> -----------------------------------------------------
>> Booting...
>> 4 vCPUs requested but only 1 available
>> -----------------------------------------------------
>> 
>> A quick look in the source code indicates that this error means bhyve can't use EPT and falls back to using 1 CPU, now the machine in question does have EPT capabilites (it lists POPCNT in dmesg):
>> 
>> -----------------------------------------------------
>> uname -a
>> FreeBSD becks.debank.tv [1] 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r253788M: Tue Jul 30 12:02:57 NZST 2013 root at becks.debank.tv:/usr/obj/usr/src/sys/GENERIC amd64
>> -----------------------------------------------------
>> -----------------------------------------------------
>> from dmesg
>> CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz (2664.82-MHz K8-class CPU)
>> Origin = "GenuineIntel" Id = 0x106a5 Family = 0x6 Model = 0x1a Stepping = 5
>> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>> Features2=0x98e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT>
>> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>> AMD Features2=0x1<LAHF>
>> TSC: P-state invariant, performance statistics
>> real memory = 25769803776 (24576 MB)
>> avail memory = 24060841984 (22946 MB)
>> -----------------------------------------------------
>> -----------------------------------------------------
>> root at becks:~/bhyve # kldstat
>> Id Refs Address Size Name
>> 1 27 0xffffffff80200000 150e0d0 kernel
>> 2 1 0xffffffff8170f000 224a98 zfs.ko
>> 3 2 0xffffffff81934000 5c30 opensolaris.ko
>> 4 1 0xffffffff8193a000 2d40 coretemp.ko
>> 5 3 0xffffffff8193d000 51a68 vboxdrv.ko
>> 6 2 0xffffffff81a12000 28c0 vboxnetflt.ko
>> 7 2 0xffffffff81a15000 881d netgraph.ko
>> 8 1 0xffffffff81a1e000 15ff ng_ether.ko
>> 9 1 0xffffffff81a20000 3ec0 vboxnetadp.ko
>> 10 1 0xffffffff81aba000 1037 cpuctl.ko
>> 11 1 0xffffffff81abc000 1686ac vmm.ko
>> -----------------------------------------------------
>> 
>> The Virtualbox logs below show EPT in use:
>> -----------------------------------------------------
>> 00:00:00.419806 HWACCM: MSR_IA32_VMX_PROCBASED_CTLS2 = 7f00000000
>> 00:00:00.419807 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_VIRT_APIC
>> 00:00:00.419807 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_EPT
>> 00:00:00.419808 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_DESCRIPTOR_INSTR_EXIT
>> 00:00:00.419808 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_RDTSCP
>> 00:00:00.419809 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_X2APIC
>> 00:00:00.419809 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_VPID
>> 00:00:00.419810 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_WBINVD_EXIT
>> ---- snip ------
>> 00:00:00.419916 HWACCM: 32-bit and 64-bit guests supported.
>> 00:00:00.419917 HWACCM: VMX enabled!
>> 00:00:00.419918 HWACCM: Enabled nested paging
>> 00:00:00.419919 HWACCM: EPT root page = 00000000b6167000
>> 00:00:00.419922 HWACCM: Large page support enabled!
>> -----------------------------------------------------
>> 
>> Thanks,
>> Rob Evers
>> 
>> _______________________________________________
>> freebsd-virtualization at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization [2]
>> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"

On 2013-08-07 10:33, Anish wrote: 
Each processor has its own EPT root pointer in VMCS. How many processors
are there in system and how many are given to Virtualbox. 

 error = vmcs_set_defaults(&vmx->vmcs[i], 
 (u_long)vmx_longjmp, 
 (u_long)&vmx->ctx[i], 
 vtophys(vmx->pml4ept), 
 pinbased_ctls, 
 procbased_ctls, 
 procbased_ctls2, 
 exit_ctls, entry_ctls, 
 vtophys(vmx->msr_bitmap), 
 vpid); 

Anish, 

The machine has one physical CPU with 4 cores using HT giving 8 threads.

I can't quickly find an option to Virtualbox to limit the amount of
cpu's used but did try to disable Virtualbox (no rc.d scripts ran and
disabled loading the kernel module) after a reboot of the machine, the
problem persists. 

-----------------------------------------------------
root at becks:~/bhyve # kldstat
Id Refs Address Size Name
 1 17 0xffffffff80200000 150e0d0 kernel
 2 1 0xffffffff8170f000 224a98 zfs.ko
 3 2 0xffffffff81934000 5c30 opensolaris.ko
 4 1 0xffffffff8193a000 2d40 coretemp.ko
 5 1 0xffffffff81a12000 1037 cpuctl.ko
 6 1 0xffffffff81a14000 1686ac vmm.ko
----------------------------------------------------- 

----------------------------------------------------- 

root at becks:~/bhyve # sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 2048
fbsd_10
Launching virtual machine "fbsd_10" ...
Consoles: userboot

FreeBSD/amd64 User boot, Revision 1.1
(root at becks.debank.tv, Tue Jul 30 11:28:33 NZST 2013)
Loading /boot/defaults/loader.conf
/boot/kernel/kernel text=0xd2f190 data=0x15de70+0x2baac0
syms=[0x8+0x14d648+0x8+0x1a9f58]

-------------- snip ------------------ 

Booting...
4 vCPUs requested but only 1 available
----------------------------------------------------- 

Could there be a problem in the CPU capability discovery code in bhyve
or is my machine not enabling EPT and Virtualbox wrongly reporting the
EPT capabilities? 

Rob Evers 

Links:
------
[1] http://becks.debank.tv
[2] http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization