How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance?
allanjude at freebsd.org
Wed Nov 12 23:46:36 UTC 2014
On 2014-11-12 17:31, Neel Natu wrote:
> Hi Tinker,
> On Wed, Nov 12, 2014 at 2:08 PM, Tinker <tinkr at openmailbox.org> wrote:
>> On 2014-11-12 23:55, Allan Jude wrote:
>>> On 2014-11-12 16:39, tinkr at openmailbox.org wrote:
>>>> In order justify giving energy to BHyVe, I need to know if it's
>>>> "future-proof" in that the 16 vCPU limit can be increased?
>>>> Please let the world know if BHyVe's 16 vCPU limit can be lifted in some
>>>> way, by configuration, patch, etc. (and if you want to, why this limit
>>>> is in place by default today).
>>>> freebsd-virtualization at freebsd.org mailing list
>>>> To unsubscribe, send any mail to
>>>> "freebsd-virtualization-unsubscribe at freebsd.org"
>>> You can increase the limit by editing sys/amd64/include/vmm.h
>>> #define VM_MAXCPU 16
>>> From what I've been told, things scale badly above 24 CPUs. They plan to
>>> solve this issue, but have not yet. If you system has enough cores to
>>> support using more than 16 per VM, you can modify the file and recompile
>>> the kernel and use as many CPUs as you want, but not much testing has
>>> happened with bigger numbers.
>> Dear Allan,
>> Thank you very much for responding.
>> Aha, great!
>> Only for completeness, do you have any particular idea about
>> * When the scaling above 24 vCPU:s will be optimized, like approx how many
>> years away is it (like 1 or more than 1)?, and
> bhyve allocates memory for the maximum number of vcpus when the
> virtual machine is created. This is simple to implement and also fits
> in nicely with bhyve's loader model where the guest loader
> (bhyveload/grub-bhyve) is separate from bhyve.
> The downside is that if VM_MAXCPU is a large number then there is a
> large memory penalty for virtual machines that only use 1 or 2 vcpus.
> Hence the desire to cap it at a smallish number.
> There are a few ways to deal with this:
> - patch the code to change VM_MAXCPU (this is what you need to do today)
> - allow the maximum vcpus to be changed via a tunable (this can be
> done in short order)
> - the limit can go away when bhyve moves to a single process model
> because we'll know the number of vcpus at VM creation time.
>> * What the technological reason for the scaling is, is it that somehow the
>> BHyVe instances on the different cores need to inter-communicate, for
>> instance that all disk and network IO is done via one single core currently?
> Actually, the performance depends more on the workload and the guest
> OS so you should definitely try how it goes for your application.
> We'll be happy to help fix any issues that come up.
>> In all cases, your response is great news, as your baseline answer that it's
>> doable and only a question of optimization and tweaking of present
>> functionality -
>> Thanks again!
Right, I forgot that at MeetBSD it was decided that making the max cpus
a sysctl was a good stop-gap measure to hold us over until the redesign
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 834 bytes
Desc: OpenPGP digital signature
More information about the freebsd-virtualization