How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance?

[Neel Natu]

Hi Tinker,

On Wed, Nov 12, 2014 at 2:08 PM, Tinker <tinkr at openmailbox.org> wrote:
> On 2014-11-12 23:55, Allan Jude wrote:
>>
>> On 2014-11-12 16:39, tinkr at openmailbox.org wrote:
>>>
>>> Hi!
>>>
>>> In order justify giving energy to BHyVe, I need to know if it's
>>> "future-proof" in that the 16 vCPU limit can be increased?
>>>
>>> Please let the world know if BHyVe's 16 vCPU limit can be lifted in some
>>> way, by configuration, patch, etc. (and if you want to, why this limit
>>> is in place by default today).
>>>
>>> Thanks!!
>>> Tinker
>>>
>>> _______________________________________________
>>> freebsd-virtualization at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>>> To unsubscribe, send any mail to
>>> "freebsd-virtualization-unsubscribe at freebsd.org"
>>
>>
>> You can increase the limit by editing sys/amd64/include/vmm.h
>>
>> #define VM_MAXCPU       16
>>
>> From what I've been told, things scale badly above 24 CPUs. They plan to
>> solve this issue, but have not yet. If you system has enough cores to
>> support using more than 16 per VM, you can modify the file and recompile
>> the kernel and use as many CPUs as you want, but not much testing has
>> happened with bigger numbers.
>
>
> Dear Allan,
>
> Thank you very much for responding.
>
> Aha, great!
>
>
> Only for completeness, do you have any particular idea about
>  * When the scaling above 24 vCPU:s will be optimized, like approx how many
> years away is it (like 1 or more than 1)?, and

bhyve allocates memory for the maximum number of vcpus when the
virtual machine is created. This is simple to implement and also fits
in nicely with bhyve's loader model where the guest loader
(bhyveload/grub-bhyve) is separate from bhyve.

The downside is that if VM_MAXCPU is a large number then there is a
large memory penalty for virtual machines that only use 1 or 2 vcpus.
Hence the desire to cap it at a smallish number.

There are a few ways to deal with this:
- patch the code to change VM_MAXCPU (this is what you need to do today)
- allow the maximum vcpus to be changed via a tunable (this can be
done in short order)
- the limit can go away when bhyve moves to a single process model
because we'll know the number of vcpus at VM creation time.

>  * What the technological reason for the scaling is, is it that somehow the
> BHyVe instances on the different cores need to inter-communicate, for
> instance that all disk and network IO is done via one single core currently?
>

Actually, the performance depends more on the workload and the guest
OS so you should definitely try how it goes for your application.
We'll be happy to help fix any issues that come up.

best
Neel

>
> In all cases, your response is great news, as your baseline answer that it's
> doable and only a question of optimization and tweaking of present
> functionality -
>
> Thanks again!
>
> Tinker
>
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscribe at freebsd.org"