Proposal for better support of hypervisors and their synthetic drivers at boot-time

[Larry Melia]

Thanks, Alexander, we'll try your approach. It seems very reasonable.


On Tue, Apr 23, 2013 at 10:48 AM, Alexander Motin <mav at freebsd.org> wrote:

> On 23.04.2013 17:07, Larry Melia wrote:
>
>> I’m on a team of developers working on synthetic device drivers for
>> FreeBSD on Hyper-V. In late March, I briefly mentioned (on this list)
>> that we're trying to get our drivers included with the FreeBSD
>> distribution. Also noted were some necessary patches to the ATA driver.
>> The changes are necessary to achieve significant performance gains by
>> replacing the native ATA driver with our synthetic storage driver
>> when when Hyper-V is detected. Alexander Motin, the maintainer of the
>> ATA code-base, however, expressed some concerns about making these
>> modifications that are so specific to Hyper-V and the AMD64 build. We
>> understand his concerns and have subsequently removed these patches from
>> our vendor branch. Our drivers are now completely independent and
>> require no changes to the kernel or other drivers. The only touch-point
>> is the common makefile for modules/drivers.
>>
>> Removing our ATA patches, on the other hand, results in a huge
>> performance loss. This is because the root file system is managed by the
>> ATA driver, which is emulated under Hyper-V. Furthermore, there may be
>> other native drivers that we wish to replace with synthetic ones at boot
>> time (e.g., mouse driver, video driver), but, there appears to be no
>> easy way to do this.Therefore, we would like to work with the developer
>> community to come up with better solution that would improve support for
>> synthetic drivers at boot-time.
>>
>> One possible solution:
>>
>> (1) Move the call to init_param1() (in sys/kern/subr_parm.c), which is
>> used for hypervisor detection, to an earlier point in the boot process.
>> Presently, it appears to be called after the ATA driver is selected,
>> which is too late in the boot process. (This was discovered after some
>> testing with the ATA driver.) Therefore, before the bus drivers and
>> native controllers are detected and selected, discovery of a host
>> hypervisor should be done first.
>>
>> (2) Extend the boot process to dynamically insert and prioritize
>> synthetic drivers over native ones. Presently, FreeBSD supports the
>> selection of generic drivers for a given device, but these can be
>> overridden when a more specific driver is available. This priority
>> scheme could be extended by first detecting the presence of a
>> hypervisor, then allowing this priority mechanism to override the native
>> drivers with their synthetic cousins (i.e., we only want to override a
>> native driver when a specific hypervisor is detected and synthetic
>> drivers for it have been configured). This could be arranged in a
>> separate, table-driven, input file or perhaps, by extending the existing
>> driver/module configuration file with additional mappings of native to
>> synthetic drivers.
>>
>> (3) Upgrade the init_param1() function (in sys/kern/subr_parm.c) to use
>> the more recent approach to hypervisor detection. This approach uses the
>> CPU-identify functions to retrieve a unique signature consisting of a
>> fixed string of ASCII characters. This was done on Linux about five
>> years. For backward compatibility, however, the existing logic would be
>> retained, but augmented with this new approach. It would also be
>> conditionally added only for x86/AMD64 builds.
>>
>> When reviewing the changes we're not looking at a lot of lines of
>> code.The difficultly lies, however, in where the changes are made.
>> Obviously, we need to work closely with the committers responsible for
>> the kernel modules we're looking to touch. Retrospectively, these
>> upgrades only bring FreeBSD up to the same level of hypervisor support
>> already present in Linux.There are other issues that we would also like
>> to discuss, but the improvements described here are on the top of the
>> list.
>>
>> We look forward to working more closely with the FreeBSD community and
>> welcome your remarks and feedback on this proposal.
>>
>
> I am sorry, I don't really understand what is your proposition, so I just
> describe my vision of the solution here, hoping it will be useful.
>
> You are right that FreeBSD has priority mechanism to select best driver
> for each device. That mechanism should be sufficient to prevent default OS
> driver from attaching to emulated ATA controller when Hyper-V environment
> is detected and paravirtual drivers are installed. To do that you should
> just create a very small dummy driver for PCI bus, implementing single
> probe() method. That probe() method should in your preferable way check
> that Hyper-V environment is active and match PCI ID of the device against
> the list of used by Hyper-V for emulated ATA controller. If both conditions
> match, the probe() method should return highest priority value to override
> any other ATA driver in system for this specific device. After that the
> dummy driver should just have empty attach() and detach() routines, not
> doing anything with the emulated device, relying on paravitrual driver to
> handle the disks.
>
> Since the dummy driver can be made machine-dependent, you may use there
> any VM detection mechanisms you like, even direct MSR access. Dummy driver
> can be made completely independent from the rest of the system, so it can
> even be loaded as kernel module without requiring any base system
> modifications. If you still prefer it for some reason (for example to not
> do some complicated checks multiple times in multiple drivers), you may
> move Hyper-V detection logic out of that driver to some single place in
> base system. But you don't need to touch OS-native ATA driver in any way.
>
> --
> Alexander Motin
>