How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance?

[Allan Jude]

On 2014-11-12 17:31, Neel Natu wrote:
> Hi Tinker,
> 
> On Wed, Nov 12, 2014 at 2:08 PM, Tinker <tinkr at openmailbox.org> wrote:
>> On 2014-11-12 23:55, Allan Jude wrote:
>>>
>>> On 2014-11-12 16:39, tinkr at openmailbox.org wrote:
>>>>
>>>> Hi!
>>>>
>>>> In order justify giving energy to BHyVe, I need to know if it's
>>>> "future-proof" in that the 16 vCPU limit can be increased?
>>>>
>>>> Please let the world know if BHyVe's 16 vCPU limit can be lifted in some
>>>> way, by configuration, patch, etc. (and if you want to, why this limit
>>>> is in place by default today).
>>>>
>>>> Thanks!!
>>>> Tinker
>>>>
>>>> _______________________________________________
>>>> freebsd-virtualization at freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>>>> To unsubscribe, send any mail to
>>>> "freebsd-virtualization-unsubscribe at freebsd.org"
>>>
>>>
>>> You can increase the limit by editing sys/amd64/include/vmm.h
>>>
>>> #define VM_MAXCPU       16
>>>
>>> From what I've been told, things scale badly above 24 CPUs. They plan to
>>> solve this issue, but have not yet. If you system has enough cores to
>>> support using more than 16 per VM, you can modify the file and recompile
>>> the kernel and use as many CPUs as you want, but not much testing has
>>> happened with bigger numbers.
>>
>>
>> Dear Allan,
>>
>> Thank you very much for responding.
>>
>> Aha, great!
>>
>>
>> Only for completeness, do you have any particular idea about
>>  * When the scaling above 24 vCPU:s will be optimized, like approx how many
>> years away is it (like 1 or more than 1)?, and
> 
> bhyve allocates memory for the maximum number of vcpus when the
> virtual machine is created. This is simple to implement and also fits
> in nicely with bhyve's loader model where the guest loader
> (bhyveload/grub-bhyve) is separate from bhyve.
> 
> The downside is that if VM_MAXCPU is a large number then there is a
> large memory penalty for virtual machines that only use 1 or 2 vcpus.
> Hence the desire to cap it at a smallish number.
> 
> There are a few ways to deal with this:
> - patch the code to change VM_MAXCPU (this is what you need to do today)
> - allow the maximum vcpus to be changed via a tunable (this can be
> done in short order)
> - the limit can go away when bhyve moves to a single process model
> because we'll know the number of vcpus at VM creation time.
> 
>>  * What the technological reason for the scaling is, is it that somehow the
>> BHyVe instances on the different cores need to inter-communicate, for
>> instance that all disk and network IO is done via one single core currently?
>>
> 
> Actually, the performance depends more on the workload and the guest
> OS so you should definitely try how it goes for your application.
> We'll be happy to help fix any issues that come up.
> 
> best
> Neel
> 
>>
>> In all cases, your response is great news, as your baseline answer that it's
>> doable and only a question of optimization and tweaking of present
>> functionality -
>>
>> Thanks again!
>>
>> Tinker
>>

Right, I forgot that at MeetBSD it was decided that making the max cpus
a sysctl was a good stop-gap measure to hold us over until the redesign
of bhyveload.

-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-virtualization/attachments/20141112/ad40e5a4/attachment.sig>