Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

[Dom]

Hi Peter,

Thanks for getting back to me. Here's the info you requested:

>> [    0.163085] acpi PNP0A03:00: host bridge window
>> [0xd000000000-0xd0100fffff] (ignored, not CPU addressable)
>
>  That one is most likely a bug in bhyve, where the space used for 64-bit
> BAR placement isn't tested against the max physaddr width of the host
> CPU.
>
>  To confirm, would you be able to report on this value on your system ?
>
> # sudo pkg install cpuid
> # cpuid | grep ^80000008
On my Intel i7-4790K CPU:

# cpuid | grep ^80000008
80000008 00003027 00000000 00000000 00000000

> The device has an MSI capability, but the nvidia driver may not use it.
> bhyve PCI passthrough requires the use of MSI/MSI-x interrupts, and
> doesn't support using legacy interrupts.
>
> This could be confirmed from the output of /proc/interrupts when
> booting Linux on the system.
Output of /proc/interrupts:

             CPU0
    0:        137   IO-APIC-edge      timer
    1:          9   IO-APIC-edge      i8042
    4:        965   IO-APIC-edge      serial
    8:          0   IO-APIC-edge      rtc0
    9:          0   IO-APIC-fasteoi   acpi
   12:        138   IO-APIC-edge      i8042
   17:          0   IO-APIC-fasteoi   snd_hda_intel
   24:          0   PCI-MSI-edge      virtio0-config
   25:       8535   PCI-MSI-edge      virtio0-req.0
   26:          0   PCI-MSI-edge      virtio1-config
   27:        123   PCI-MSI-edge      virtio1-input.0
   28:          1   PCI-MSI-edge      virtio1-output.0
NMI:          0   Non-maskable interrupts
LOC:       6050   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:       2484   IRQ work interrupts
RTR:          0   APIC ICR read retries
RES:          0   Rescheduling interrupts
CAL:          0   Function call interrupts
TLB:          0   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          0   Machine check exceptions
MCP:          1   Machine check polls
ERR:          0
MIS:          0

I guess the lack of a line containing PCI-MSI-* here indicates the 
nvidia driver isn't using an MSI/MSI-x interrupt?
However, searching the web suggests the Linux nvidia driver does use MSI 
interrupts. This taken from a working non-VM Linux dmesg:

[    4.330536] nvidia 0000:05:00.0: PCI INT A -> GSI 16 (level, low) -> 
IRQ 16
[    4.330542] nvidia 0000:05:00.0: setting latency timer to 64

Source: https://bugzilla.kernel.org/show_bug.cgi?id=20432#c2
(Thread also mentions disabling MSI)

I'll try some Linux boot options and reordering the devices when calling 
bhyve to see if that changes anything.

Thanks,

Dom