High vCPU Counts in bhyve

[Trent Thompson]

Good day to all!

I was recently given the hint that I could edit vmm.h to allow for more
than 16 vCPUs in bhyve, so I did just that to see what would happen. I
understand it's not supported, just wanted to try it out. When trying to
boot FreeBSD 12-CURRENT with anything up to 25 vCPUs the guest will boot
and run just fine. At 26+ vCPUs the guest OS will panic at boot time.

I have posted all of these outputs and more to GitHub [1].

I started by installing FreeBSD with 16 vCPUs and then incrementing until
it wouldn't go any further. You can find the dmesg.boot output of the guest
at 16 vCPUs on GitHub [2].

Here is the relevant line in /usr/src/sys/amd64/include/vmm.h:

#define VM_MAXCPU    32    /* maximum virtual cpus */

Here is what appears in the serial console when attempting to boot with 26
vCPUs (The full output can also be found on GitHub [3]):

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x60
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff80ff3859
stack pointer           = 0x28:0xffffffff821c19d0
frame pointer           = 0x28:0xffffffff821c1a50
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (swapper)
[ thread pid 0 tid 100000 ]
Stopped at      acpi_install_wakeup_handler+0x1d9:      movq    %rdi,0x60(%r14)
db>


If anyone has any hints on how to get more information from this or ideas
on the apparent APIC error would be greatly appreciated.

Thanks and happy hacking!

-Trent
@pr1ntf

[1] https://gist.github.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31

[2]
https://gist.githubusercontent.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31/raw/09dde8362a5e488794e6d7c8b70bb0368b964b49/3-guest-dmesg.boot.txt

[3]
https://gist.githubusercontent.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31/raw/09dde8362a5e488794e6d7c8b70bb0368b964b49/4-guest-26vcpucrash.txt