Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

[Peter Grehan]

Hi,

>>   That's correct - it's a bug in bhyve.
>
> Baking a proper fix will be more complicated by the fact that PCIe
> cards themselves may have limitations. For example, most nVidia GPUs
> have 40 bits DMA addressing capability, some 39, an a few (still
> modern) ones -- just 37 [ref. nVidia "README" in the driver package].

  That's a different issue - it's unlikely, if not impossible, to 
configure bhyve with enough RAM to hit 37 bits worth where that would 
become a problem. No need to worry about that.

>>   PCI passthru doesn't allow the BAR values to be modified (this could
>> be changed, but it's a lot of work for little gain).
>
> Removing another signature of detecting virtualization and increasing
> compatibility would be negligible gain? Just asking...

  There are lots of BIOS/UEFI implementations out there that have the 
same restriction. In general, there should be no need for a guest to 
reprogram device BARs.

  After changing the 64-bit BAR base address, did you still need the 
pci=nocrs option for Linux ? I'd hope this would be no longer necessary.

>>> But:
>>>   # ./nvidia-smi
>>>   No devices were found
>>> dmesg:
>>>   [  173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856)
>>>   [  173.499115] NVRM: rm_init_adapter failed for device bearing
>>> minor number 0
>>
>>   Looks like you're getting close :)
>
> Hmm, I'm not seeing myself getting much closer here. Do you know
> something I don't? ;) I really hope bhyve developers can spare a
> bit of time on getting GPU passthrough to work... I know nothing
> about these things, and where I waste half a day messing around,
> the problem could be fixed in half an hour by someone who knows.

  The problem is the knowledge set of graphics/GPU knowhow and equipment 
access, and bhyve/PCI programming, are disjoint. The time I've spent on 
it has been the inverse, where I feel that I've spent a half-day doing 
things that anyone who knew about graphics could get done in a half-hour :)

  For these type of issues, joint work is best to leverage the knowledge 
of both sides. From my point-of-view, the work you've done has been very 
helpful.

later,

Peter.