bhyve PCIe passthrough on AMD Epyc

[Anish]

Hi Kevin,
>I'm just pointing out that it's an AMD Epyc because I know the AMD
IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is
maybe less tested.
I tested on Ryzen-I, not sure if anyone tried it on newer Ryzen/Epyc
systems.

>CPU0: local APIC error 0x40
This is concerning, as per AMD docs, bit6 is for illegal vector which
indicate CPU is getting invalid interrupts.

Can you share hw.vmm sysctl output?

-Anish


On Thu, Feb 28, 2019 at 9:59 AM Kevin Day <toasty at dragondata.com> wrote:

>
>
> > On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes <
> freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
> >
> >>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes <
> freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
> >>>
> >>>> I'm trying to get PCIe device (video card) passthrough working to a
> Windows bhyve VM, should this work now?
> >>>
> >>> No, this should not work now.
> >>
> >> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now?
> > I do not know what would be special about Epyc so can not say
> > for certain.
>
> I'm just pointing out that it's an AMD Epyc because I know the AMD
> IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is
> maybe less tested.
>
> >> Just trying the same thing with a simple PCIe USB card gives me:
> >>
> >> CPU0: local APIC error 0x40
> >> CPU0: local APIC error 0x40
> >> ivhd0: Error: completion failed tail:0xda0, head:0x0.
> >> ivhd0: Dump all the commands:
> >>  [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0
> >>  [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003
> >>  [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0
> >>  [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0
> >>  [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0
> >> [...snip... lots of similar stuff]
> >>
> >> Then I get a bunch more APIC errors, and the system freezes for about
> 60 seconds before recovering.
> >
> > You do not get any host dmesg errors about unaligned BAR
> > ranges or sizes?
> >
>
> Correct. I did with the video card, but not with the xhci USB card. I get
> a ton of dmesg errors, and the host system repeatedly freezes until I stop
> the VM. (Console is non-responsive, no ICMP replies, etc)
>
> I didn't actually look at this until now, but Windows actually does see
> the USB controller, but it complains that "port reset failed" on any device
> I attach to it.
>
> > Can you get the pciconf -l -b info for the device you passed, something
> like:
> > # pciconf -l -b uhci0
> > uhci0 at pci0:0:26:0:      class=0x0c0300 card=0x20f017aa chip=0x29378086
> rev=0x03 hdr=0x00
> >    bar   [20] = type I/O Port, range 32, base 0x1840, size 32, enabled
>
> xhci3 at pci0:97:0:0:      class=0x0c0330 card=0xffffffff chip=0x00151912
> rev=0x02 hdr=0x00
>     bar   [10] = type Memory, range 64, base 0xe1d00000, size 8192, enabled
>     cap 01[50] = powerspec 3  supports D0 D3  current D0
>     cap 05[70] = MSI supports 8 messages, 64 bit enabled with 1 message
>     cap 11[90] = MSI-X supports 8 messages
>                  Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
>     cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
>                  link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1)
>     ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
>     ecap 0018[150] = LTR 1
>
> Also just in case it's relevant, "acpidump" on this system hangs after
> allocating ~3GB of ram and just spins at 100% CPU. I am seeing that AMD-Vi
> is enabled in the boot dmesg though:
>
> AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0
>
>
>
>
>
>
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscribe at freebsd.org"
>