bhyve PCIe passthrough on AMD Epyc

[Kevin Day]

> On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes <freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
> 
>>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes <freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
>>> 
>>>> I'm trying to get PCIe device (video card) passthrough working to a Windows bhyve VM, should this work now?
>>> 
>>> No, this should not work now.
>> 
>> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now?
> I do not know what would be special about Epyc so can not say
> for certain.

I'm just pointing out that it's an AMD Epyc because I know the AMD IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is maybe less tested.

>> Just trying the same thing with a simple PCIe USB card gives me:
>> 
>> CPU0: local APIC error 0x40
>> CPU0: local APIC error 0x40
>> ivhd0: Error: completion failed tail:0xda0, head:0x0.
>> ivhd0: Dump all the commands:
>>  [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0
>>  [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003
>>  [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0
>>  [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0
>>  [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0
>> [...snip... lots of similar stuff]
>> 
>> Then I get a bunch more APIC errors, and the system freezes for about 60 seconds before recovering.
> 
> You do not get any host dmesg errors about unaligned BAR
> ranges or sizes?
> 

Correct. I did with the video card, but not with the xhci USB card. I get a ton of dmesg errors, and the host system repeatedly freezes until I stop the VM. (Console is non-responsive, no ICMP replies, etc)

I didn't actually look at this until now, but Windows actually does see the USB controller, but it complains that "port reset failed" on any device I attach to it.

> Can you get the pciconf -l -b info for the device you passed, something like:
> # pciconf -l -b uhci0
> uhci0 at pci0:0:26:0:      class=0x0c0300 card=0x20f017aa chip=0x29378086 rev=0x03 hdr=0x00
>    bar   [20] = type I/O Port, range 32, base 0x1840, size 32, enabled

xhci3 at pci0:97:0:0:	class=0x0c0330 card=0xffffffff chip=0x00151912 rev=0x02 hdr=0x00
    bar   [10] = type Memory, range 64, base 0xe1d00000, size 8192, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit enabled with 1 message
    cap 11[90] = MSI-X supports 8 messages
                 Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
                 link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0018[150] = LTR 1

Also just in case it's relevant, "acpidump" on this system hangs after allocating ~3GB of ram and just spins at 100% CPU. I am seeing that AMD-Vi is enabled in the boot dmesg though:

AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0