problem with pass-through on amd

[Anish]

Hi Andriy,

>What's suspicious is that there are no interrupts either for the
ppt device on the host or the ahci device in the guest.

I think most likely AMD IOMMU/amdvi is not enabled, by default it is
disabled and can be confirmed by sysctl.

[root at ryzen /usr/home/anish]# sysctl hw.vmm.amdvi
hw.vmm.amdvi.domain_id: 3
hw.vmm.amdvi.disable_io_fault: 0
hw.vmm.amdvi.ptp_level: 4
hw.vmm.amdvi.host_ptp: 1
hw.vmm.amdvi.enable: 1
hw.vmm.amdvi.count: 2

If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work.

Regards,
Anish


On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <avg at freebsd.org> wrote:

>
> I have a FreeBSD guest on a FreeBSD host, both amd64 head.
> I decided to experiment with PCI pass-through and used an AHCI controller
> for
> the experiment.  It seems like that partially works.  The guest detects the
> controller and attaches a driver to it, the messages (reported
> capabilities,
> etc) look identical to those on the host.  But the guest can not detect any
> disks behind the controller.  From what I can tell, the guest has no
> problems
> accessing PCI configuration registers and a memory mapped BAR.  Not sure
> about
> the I/O ports.  What's suspicious is that there are no interrupts either
> for the
> ppt device on the host or the ahci device in the guest.  Maybe that is what
> causes the failure to see the disks.
>
> When the guest driver attaches to the controller I see these messages on
> the host:
> ppt0: attempting to allocate 1 MSI vectors (1 supported)
> ppt0: using IRQ 265 for MSI
> and in the guest:
> ahci0: attempting to allocate 1 MSI vectors (1 supported)
> ahci0: using IRQ 279 for MSI
>
> But vmstat -i does not report any interrupts.
>
> Just in case, the controller has some complex topology where the AHCI PCI
> device
> is behind two PCI-PCI bridges from the main bus.  But I think that that
> should
> not matter and the guest needs to see only the AHCI device and the host
> handles
> the bridges.
>
> Here are some more messages from the guest:
> ahci0: <HighPoint RocketRAID 640 AHCI SATA controller> port
> 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem
> 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0
> ahci0: attempting to allocate 1 MSI vectors (1 supported)
> ahci0: using IRQ 279 for MSI
> ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
> ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports
> ahcich0: <AHCI channel> at channel 0 on ahci0
> ahcich0: Caps: CPD ESP FBSCP
> ahcich1: <AHCI channel> at channel 1 on ahci0
> ahcich1: Caps: CPD ESP FBSCP
> ahcich0: AHCI reset...
> ahcich0: SATA connect time=100us status=00000123
> ahcich0: AHCI reset: device found
> ahcich1: AHCI reset...
> ahcich1: SATA connect time=100us status=00000123
> ahcich1: AHCI reset: device found
> ahcich1: AHCI reset: device ready after 100ms
> ahcich1: Poll timeout on slot 1 port 15
> ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr
> 00000000
> cmd 00318017
> ahcich0: AHCI reset: device ready after 200ms
> ahcich0: Poll timeout on slot 1 port 15
> ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr
> 00000000
> cmd 00318017
> And some more...
>
> Any ideas or suggestions?
> Or maybe some additional information from me?
>
> Thank you!
>
> --
> Andriy Gapon
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-
> unsubscribe at freebsd.org"
>