Checking bhyve supported features (sysctls)

[Matt Churchyard]

-----Original Message-----
From: owner-freebsd-virtualization at freebsd.org <owner-freebsd-virtualization at freebsd.org> On Behalf Of Rodney W. Grimes
Sent: 16 August 2018 18:31
To: Allan Jude <allanjude at FreeBSD.org>
Cc: Matt Churchyard <matt.churchyard at userve.net>; freebsd-virtualization at freebsd.org
Subject: Re: Checking bhyve supported features (sysctls)

> On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" <freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
> >> 
> >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> >> 
> >> > > Hello,
> >> > > 
> >> > > I'm looking for better ways to check for  bhyve support /
> >available
> >> > > features without trying to scan through dmesg output.
> >> > 
> >> > >Yes, it would be very good to remove that, as it usually tries 
> >> > >to grep a non-existent file /var/run/dmesg.boot that is not 
> >> > >created until after vm_bhyve has been called from
> >/usr/local/etc/rc.d
> >> > >when you have things set to autostartup >in /etc/rc.conf
> >> > 
> >> > 
> >> > > 
> >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> >soon
> >> > > as the vmm module is loaded
> >> > > 
> >> > > hw.vmm.vmx.initialized: 1
> >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> >> > > 
> >> > > Will these be available on both Intel & AMD processors as a way 
> >> > > to determine if the module has loaded successfully and can run
> >guests?
> >> > > 
> >> > > I also see the below sysctl related to iommu.
> >> > > 
> >> > > hw.vmm.iommu.initialized
> >> > > 
> >> > > Again, will this be set to 1 as soon as the module is loaded if 
> >> > > iommu is supported, or only when it is used?
> >> > > There also seems to be a vmm.amdvi.enable sysctl.
> >> > > Would both these need checking or is vmm.iommu enough to 
> >> > > determine support on any processor.
> >> > 
> >> > >Probalby the safest way for a shell script to decide if bhyve is 
> >> > >up and running is to stat /dev/vmm, if that exists then the
> >modules
> >> > >have loaded and initialized and bhyve should be ready to process
> >guests.
> >> > 
> >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> >> 
> >> I'll investigate that, I was pretty sure that you should get this 
> >> as soon as the vmm.ko module is finished initialzing, but you might 
> >> be right in that it takes a first vm to cause its creation.
> >> Confirmed, /dev/vmm does not exist until the first vm is created.
> >> 
> >> > 
> >> > >sysctl's mentiond above would be a poor way to make this
> >determination.
> >> > 
> >> > It would be nice if sysctls were better documented.
> >> 
> >> Agreed.
> >> 
> >> > If vmx.initialized is set once vmm has successfully loaded, I 
> >> > can't
> >see a better way of checking for bhyve support (assuming it's not 
> >Intel specific). This entry definitely exists and is set to 0 if you 
> >load the module on a non-supported system, and set to 1 as soon as 
> >vmm loads on my Intel test system.
> >> 
> >> Given its undocumented status you would be relying on an 
> >> undocumented feature that could change in either name or behavior, 
> >> and that is not desirable.
> >> 
> >> Let me see if I can come up with something else.
> >
> >I looked at the code for bhyvectl, bhyveload and byhve.  They do not 
> >actually try to decide if vmm is supported or not, they simply 
> >process the error from a vm_create() or vm_open() call and exit with 
> >an error code if they can not handle it (some of the code can handle 
> >a vm_create failure if infact we are trying to create a vm that 
> >already exists).
> >
> >If you want to maintain full compatibility a similiar stratergy may 
> >be in order.
> >
> >Why is it that vm-bhyve specifically needs to know if the kernel has 
> >vmm support or not?
> >Cant it just be written to handle the errors returned if the 
> >supported functions do not exist?
> 
> I think the question vm-bhyve wants to answer is: does the CPU have 
> the required features to run a multicore VM.

>Why does it need to know that?  If it tries to start a multicore/thread VM and the system can not support it an error is returned and the bhyve command fails.

>Actually determining that specific issue is non-trivial iirc as some vmm supported CPU's can only run a single VM with a single thread in that VM (early core cpu's).

> 
> These or similar sysctls do seem to be the correct way to communicate 
> that support.

>I do not believe any of those sysctl's communicate that your on a broken cpu or to what extent you can run vm's with multiple threads.

So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is not a valid way to determine if the host has unrestricted guest support (required for non-freebsd or multicore freebsd guests, and as you say missing from some early VT-x capable processors)?

>I went and looked at why vm-bhyve is groveling around in /var/run/dmesg.boot and found that it is simply trying to determine if the host CPU is vmm capable,
>specifically:
>util::check_bhyve_support(){
>...

>These checks are already built into the kernel.
>This can all go in the bit bucket, if you try to start a VM on an unsupported system an error is returned, recoding this in shell is just setting yourself up for "future" bugs.

The kernel knows what features are supported but does not expose these, so all I can do is similar to libvirt and run bhyve with different options to see what errors pop up.
I think I'll just remove all checking for now and let users discover the issue for themselves if bhyve won't run. Hopefully the vmx.initialized / cap.* sysctls will at some point become a defined way of seeing if vmm is ready / testing for vmm features, as apparently these serve no purpose at the moment.

Matt