increasing bhyve VM_MAXCPU
John Doherty
bsdlists at jld3.net
Tue May 18 20:51:40 UTC 2021
I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD
12.2-RELEASE.
As I understand it, this limit is defined as VM_MAXCPU in
/usr/src/sys/amd64/include/vmm.h. I also understand that as of about two
years ago, it should be possible to increase it beyond 21[1].
I began working on a machine with only eight CPU cores/threads, so
rather than trying to increase VM_MAXCPU, I tried to decrease it, which
seemed to work fine. My procedure was:
edit /usr/src/sys/amd64/include/vmm.h
cd /usr/src/lib/libvmmapi ; make clean ; make ; make install
cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install
cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install
cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install
cd /usr/src ; make buildkernel KERNCONF=JLD ; make installkernel
KERNCONF=JLD
reboot
where JLD is a kernel config file that differs from GENERIC only in the
"ident" line.
I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs
with a number of vCPUs less than or equal to the max booted fine and
that VMs with more didn't.
So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14
cores/28 threads each, using the same procedure. I set VM_MAXCPU to 28
and configured a VM with 24 vCPUs but that didn't boot. Also tried
setting the max to 20 and booting a VM with 18 vCPUs but that didn't
work either. In all cases, VMs with 16 or fewer vCPUs continued to boot
fine.
All the tests were done with VMs created before doing any of this stuff.
I am using vm-bhyve to manage VMs. The VMs in these tests were running
one of FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing.
So it seems like I'm overlooking something but I don't know what, or
maybe this just isn't going to work. Anybody have experience actually
doing this?
Thanks very much for any advice or further information.
[1] <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212782>
More information about the freebsd-virtualization
mailing list