Re: bhyve vCPU limit

On 01/12/2021 17:17, John Doherty via freebsd-virtualization wrote:
> That limitation appears to still exist in FreeBSD 13.0-RELEASE:
> 
> [root@grit] # freebsd-version -k ; grep 'VM_MAXCPU' 
> /usr/src/sys/amd64/include/vmm.h
> 13.0-RELEASE
> #define    VM_MAXCPU    16            /* maximum virtual cpus */
> 
> I ran into this in May 2021 and with some help from folks on this list 
> was able to increase it. The simplest (if not minimalist) way to do that 
> is:
> 
> 1. edit /usr/src/sys/amd64/include/vmm.h to increase that value: I used 48
> 2. make buildworld
> 3. make installworld
> 
> The increased value has been working fine for me since I did that. I run 
> a couple of VMs with 24 vCPUs each and several others with smaller 
> numbers all the time and have run others with as many as 48 temporarily. 
> No problems that I have seen.

I am sorry for hijacking this thread but your information is very 
interesting. I was playing with VMs in VirtualBox and Bhyve and compared 
performance with increasing vCPU count. The more cores VM get the slower 
was even a simple single threaded task like loading PF rules from 
/etc/pf.conf. It was tested on FreeBSD 11.4 and 12.2, I tested ULE and 
4BSD schedulers. Maybe it was somewhat HW related but it always shows 
VMs with more than 2 v CPUs significantly slower. VMs with 6+ vCPU was 
almost unusable (loading of PF ruleset takes about 8 seconds instead of 
fraction on single vCPU VM).

Do you have any special tunning to have so large number of vCPU without 
this penalty?

Kind regards
Miroslav Lachman