Checking bhyve supported features (sysctls)

[Rodney W. Grimes]

> On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" <freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
> >> 
> >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> >> 
> >> > > Hello,
> >> > > 
> >> > > I'm looking for better ways to check for  bhyve support /
> >available
> >> > > features without trying to scan through dmesg output.
> >> > 
> >> > >Yes, it would be very good to remove that, as it usually tries
> >> > >to grep a non-existent file /var/run/dmesg.boot that is not
> >> > >created until after vm_bhyve has been called from
> >/usr/local/etc/rc.d
> >> > >when you have things set to autostartup >in /etc/rc.conf
> >> > 
> >> > 
> >> > > 
> >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> >soon 
> >> > > as the vmm module is loaded
> >> > > 
> >> > > hw.vmm.vmx.initialized: 1
> >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> >> > > 
> >> > > Will these be available on both Intel & AMD processors as a way
> >> > > to determine if the module has loaded successfully and can run
> >guests?
> >> > > 
> >> > > I also see the below sysctl related to iommu.
> >> > > 
> >> > > hw.vmm.iommu.initialized
> >> > > 
> >> > > Again, will this be set to 1 as soon as the module is loaded if
> >> > > iommu is supported, or only when it is used?
> >> > > There also seems to be a vmm.amdvi.enable sysctl.
> >> > > Would both these need checking or is vmm.iommu enough to
> >> > > determine support on any processor.
> >> > 
> >> > >Probalby the safest way for a shell script to decide if bhyve is
> >> > >up and running is to stat /dev/vmm, if that exists then the
> >modules
> >> > >have loaded and initialized and bhyve should be ready to process
> >guests.
> >> > 
> >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> >> 
> >> I'll investigate that, I was pretty sure that you should get this
> >> as soon as the vmm.ko module is finished initialzing, but you might
> >> be right in that it takes a first vm to cause its creation.
> >> Confirmed, /dev/vmm does not exist until the first vm
> >> is created.
> >> 
> >> > 
> >> > >sysctl's mentiond above would be a poor way to make this
> >determination.
> >> > 
> >> > It would be nice if sysctls were better documented.
> >> 
> >> Agreed.
> >> 
> >> > If vmx.initialized is set once vmm has successfully loaded, I can't
> >see a better way of checking for bhyve support (assuming it's not Intel
> >specific). This entry definitely exists and is set to 0 if you load the
> >module on a non-supported system, and set to 1 as soon as vmm loads on
> >my Intel test system.
> >> 
> >> Given its undocumented status you would be relying on an
> >> undocumented feature that could change in either name or
> >> behavior, and that is not desirable.
> >> 
> >> Let me see if I can come up with something else.
> >
> >I looked at the code for bhyvectl, bhyveload and
> >byhve.  They do not actually try to decide if vmm
> >is supported or not, they simply process the error
> >from a vm_create() or vm_open() call and exit
> >with an error code if they can not handle it
> >(some of the code can handle a vm_create failure
> >if infact we are trying to create a vm that
> >already exists).
> >
> >If you want to maintain full compatibility a similiar
> >stratergy may be in order.
> >
> >Why is it that vm-bhyve specifically needs to know
> >if the kernel has vmm support or not?
> >Cant it just be written to handle the errors returned
> >if the supported functions do not exist?
> 
> I think the question vm-bhyve wants to answer is: does
> the CPU have the required features to run a multicore VM.

Why does it need to know that?  If it tries to start a multicore/thread
VM and the system can not support it an error is returned and the bhyve
command fails.

Actually determining that specific issue is non-trivial iirc as
some vmm supported CPU's can only run a single VM with a single
thread in that VM (early core cpu's).

> 
> These or similar sysctls do seem to be the correct way
> to communicate that support.

I do not believe any of those sysctl's communicate that
your on a broken cpu or to what extent you can run vm's
with multiple threads.


I went and looked at why vm-bhyve is groveling around
in /var/run/dmesg.boot and found that it is simply
trying to determine if the host CPU is vmm capable,
specifically:
util::check_bhyve_support(){
...
    # get features2 line which should include popcnt
    _mesg=$(grep -E '^[ ]+Features2' /var/run/dmesg.boot | tail -n 1)

    # only check if we found it
    if [ -n "${_mesg}" ]; then

        # look for pop cnt
        _result=$(echo "${_mesg}" |grep "POPCNT")
        [ -z "${_result}" ] && util::err "it doesn't look like your cpu supports bhyve (missing POPCNT)"
    fi

    # check ept for intel
    _mesg=$(grep -E '^[ ]+VT-x' /var/run/dmesg.boot | tail -n 1)

    if [ -n "${_mesg}" ]; then

        # look for ept
        _result=$(echo "${_mesg}" |grep "EPT")
        [ -z "${_result}" ] && util::err "it doesn't look like your cpu supports bhyve (missing EPT)"

        # look for unrestricted guest
        _result=$(echo "${_mesg}" |grep ",UG")
        [ -z "${_result}" ] && VM_NO_UG="1"
    fi

These checks are already built into the kernel.
This can all go in the bit bucket, if you try to start
a VM on an unsupported system an error is returned,
recoding this in shell is just setting yourself
up for "future" bugs.

-- 
Rod Grimes                                                 rgrimes at freebsd.org