Bhyve GPU passthrough

[Robert Crowston]

I think I am the most recent adventurer, but if you go back a couple of years I am only building on material I found discussed in this mailing list.

I do have an nVidia GPU working on Linux under the nouveau drivet. It required some changes to the bhyve source; in particular I had to remap the 64 bit PCI BAR allocations into the lower 4 GB of the guest’s address space. I am not sure if that is a quirk of the AMD processor I’m using.

I think there are a few things that prevent this from working in the general case. Probably the most serious is bhyve doesn’t support emulation or passthrough of PCI ROMs. I am working on a hack to make that work in my spare time. But I am not an official member of the project, just tinkering. :-)

The nVidia Windows driver only checks the pending break enable bit on the feature flags register to see if it is virtualized (or at least that was the case about a year ago). You can readily change that to 0 in the bhyve source code. But we are some way from that yet.

On Fri, Jun 14, 2019 at 20:53, Adam <amvandemore at gmail.com> wrote:

> There was a thread here a while back about some adventurer trying to
> passthrough an Nvidia GPU which ultimately did not work. Unfortunately
> that still seems to be the case. I'm using 12.x and I could not get it to
> work in FreeBSD, Linux, or Windows guest. Win10 wouldn't even boot with it
> passed through. Linux wouldn't load the driver(could have been me) and
> FreeBSD guest loaded the driver, but I could not do anything useful with
> the card. For example, nvidia-smi -q reported No devices were found. Here
> is the guest view:
>
> vgapci0 at pci0:0:1:0: class=0x030000 card=0x77101558 chip=0x1f5010de
> rev=0xa1 hdr=0x00
> vendor = 'NVIDIA Corporation'
> device = 'TU106BM [GeForce RTX 2070 Mobile]'
> class = display
> subclass = VGA
> bar [10] = type Memory, range 32, base 0xc0000000, size 16777216,
> enabled
> bar [14] = type Prefetchable Memory, range 64, base 0xd000000000,
> size 268435456, enabled
> bar [1c] = type Prefetchable Memory, range 64, base 0xc2000000, size
> 33554432, enabled
> bar [24] = type I/O Port, range 32, base 0x2000, size 128, enabled
> cap 01[60] = powerspec 3 supports D0 D3 current D0
> cap 05[68] = MSI supports 1 message, 64 bit enabled with 1 message
> cap 10[78] = PCI-Express 2 legacy endpoint max data 128(256) FLR RO NS
> link x16(x16) speed 2.5(8.0) ASPM disabled(L0s/L1)
>
> I believe in earlier discussion there was objection to overcoming Nvidia's
> virtualization restrictions, but I think since then both vmware and kvm
> have implemented workarounds to allow for this.
>
> Is anyone one currently working on this or needs some testing done?
>
> --
> Adam
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"