increasing bhyve VM_MAXCPU

[Shawn Webb]

On Tue, May 18, 2021 at 02:51:22PM -0600, John Doherty via freebsd-virtualization wrote:
> I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD
> 12.2-RELEASE.
> 
> As I understand it, this limit is defined as VM_MAXCPU in
> /usr/src/sys/amd64/include/vmm.h. I also understand that as of about two
> years ago, it should be possible to increase it beyond 21[1].
> 
> I began working on a machine with only eight CPU cores/threads, so rather
> than trying to increase VM_MAXCPU, I tried to decrease it, which seemed to
> work fine. My procedure was:
> 
>   edit /usr/src/sys/amd64/include/vmm.h
>   cd /usr/src/lib/libvmmapi ; make clean ; make ; make install
>   cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install
>   cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install
>   cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install
>   cd /usr/src ; make buildkernel KERNCONF=JLD ; make installkernel
> KERNCONF=JLD
>   reboot
> 
> where JLD is a kernel config file that differs from GENERIC only in the
> "ident" line.
> 
> I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs
> with a number of vCPUs less than or equal to the max booted fine and that
> VMs with more didn't.
> 
> So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14 cores/28
> threads each, using the same procedure. I set VM_MAXCPU to 28 and configured
> a VM with 24 vCPUs but that didn't boot. Also tried setting the max to 20
> and booting a VM with 18 vCPUs but that didn't work either. In all cases,
> VMs with 16 or fewer vCPUs continued to boot fine.
> 
> All the tests were done with VMs created before doing any of this stuff. I
> am using vm-bhyve to manage VMs. The VMs in these tests were running one of
> FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing.
> 
> So it seems like I'm overlooking something but I don't know what, or maybe
> this just isn't going to work. Anybody have experience actually doing this?
> 
> Thanks very much for any advice or further information.

HardenedBSD bumped max vCPUs up to 48 in a recent commit[0]. Our
14-CURRENT/amd64 package building VM is set up with 30 (out of 36
available) vCPUs (`vmrun.sh -c 30`). We've completed one full package
build (so, 31,000+ ports) without issue. The one package building VM
we have on the system (so no other VMs) is running HardenedBSD
14-CURRENT. Meaning, we don't run anything other than HardenedBSD VMs.

[0]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/issues/30

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-virtualization/attachments/20210518/3457647a/attachment.sig>