Run bhyve in KVM on Intel E5-2660 v4 issues

[Elena Mihailescu]

Hello,

We have some issues on using bhyve from inside a FreeBSD KVM virtual
machine on Intel E5-2660 v4 servers.  The host operating systems we
tried and the errors are described below.

If the host is one of the following operating systems: Ubuntu 18.04,
Ubuntu 18.04 for servers or CentOS8 we get the following behavior:
Tested on FreeBSD-Current
(FreeBSD-13.0-CURRENT-amd64-20200213-r357847) and FreeBSD12.1:
- starting bhyve using bhyveload as a loader ends with a kernel panic
in the bhyve VM due to APIC error:
# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1G -d guest.img -i -I
FreeBSD-12.1-RELEASE-amd64-disc1.iso machine
====
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE  BVMADT >
panic: APIC: CPU with APIC ID 255 is not enabled
cpuid = 0
time = 1
KDB: stack backtrace:
#0 0xffffffff80c1d297 at kdb_backtrace+0x67
#1 0xffffffff80bd05cd at vpanic+0x19d
#2 0xffffffff80bd0423 at panic+0x43
#3 0xffffffff811f7569 at madt_set_ids+0x99
#4 0xffffffff80b66918 at mi_startup+0x118
#5 0xffffffff8036b02c at btext+0x2c
Uptime: 1s
====

- starting bhyve using uefi as a loader seems to stuck or run in an
infinite loop  - nothing is displayed on screen after starting the VM,
we cannot interact with it.

bhyve -c 1 -m 1G -H -P -S -s 0:0,hostbridge -s 1:0,lpc -s
4:0,ahci-hd,guest.img -s 3:0,ahci-cd,<ISO> -l com1,stdio -l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd <vmname>
*stuck - does not receive input, does not display anything.

We also tried disabling x2apic from the host kernel options and also
from the KVM guest options. As for libvirt options, we tried both cpu
options: host-model and host-passthrough.

If the host is a CentOS7, we didn't manage to get the FreeBSD KVM
virtual machine to see the hardware virtualization features and we
cannot run `kldload vmm` successfully (error: vmx_init: processor does
not support desired basic capabilities). We have tested with a CentOS7
guest and nested virtualization with kvm does not have any issues.

In the document at [1], we centralized the tests we run and their
output, including virsh xml files, bhyve commands, and the errors we
have encounters. As a parallel, we installed FreeBSD directly on the
server and ran the same bhyve commands that worked fine.

Does anyone know any way to solve this issue?

[1]  https://docs.google.com/document/d/1DIzdJKHNZNLZXk9dJquUDwwYJpTH2zDkT9eF2sVIb-A/edit?usp=sharing

Thank you,
Elena