problem with pass-through on amd

[Andriy Gapon]

On 13/11/2017 22:41, Anish wrote:
> Hi Andriy,
> 
>>What's suspicious is that there are no interrupts either for the
> ppt device on the host or the ahci device in the guest.
> 
> I think most likely AMD IOMMU/amdvi is not enabled, by default it is disabled
> and can be confirmed by sysctl.
> 
> [root at ryzen /usr/home/anish]# sysctl hw.vmm.amdvi
> hw.vmm.amdvi.domain_id: 3
> hw.vmm.amdvi.disable_io_fault: 0
> hw.vmm.amdvi.ptp_level: 4
> hw.vmm.amdvi.host_ptp: 1
> hw.vmm.amdvi.enable: 1
> hw.vmm.amdvi.count: 2 
> 
> If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work.


I followed your how-to, so I didn't make this mistake :-)

$ sysctl hw.vmm.amdvi
hw.vmm.amdvi.domain_id: 3
hw.vmm.amdvi.disable_io_fault: 0
hw.vmm.amdvi.ptp_level: 4
hw.vmm.amdvi.host_ptp: 1
hw.vmm.amdvi.enable: 1
hw.vmm.amdvi.count: 1

Could it be that AMD-Vi support is broken in the hardware or BIOS?
Are there any checks that I could do?

Here is what I see in dmesg, but it is not a full dump of IVRS, of course:
$ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu'
Table 'IVRS' at 0xbdafaf80
ACPI: IVRS 0x00000000BDAFAF80 0000F8 (v01 AMD    RD890S   00202031 AMD  00000000)
AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0
ivhd0: <AMD-Vi/IOMMU or ivhd> on acpi0
ivhd0: Unknown dev entry:0xff
ivhd0: Flag:3e<PassPW,ResPassPW,Isoc,IotlbSup,Coherent>
ivhd0: max supported paging level:7 restricting to: 4
ivhd0: device supported range [0x0 - 0xb01]
ivhd0: device [0xa0 - 0xa0]config:d7<INIT,ExtInt,NMI,LINT0>
ivhd0: PCI cap 0x10b540f at 0x40 feature:1<IOTLB>
pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)

> On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <avg at freebsd.org
> <mailto:avg at freebsd.org>> wrote:
> 
> 
>     I have a FreeBSD guest on a FreeBSD host, both amd64 head.
>     I decided to experiment with PCI pass-through and used an AHCI controller for
>     the experiment.  It seems like that partially works.  The guest detects the
>     controller and attaches a driver to it, the messages (reported capabilities,
>     etc) look identical to those on the host.  But the guest can not detect any
>     disks behind the controller.  From what I can tell, the guest has no problems
>     accessing PCI configuration registers and a memory mapped BAR.  Not sure about
>     the I/O ports.  What's suspicious is that there are no interrupts either for the
>     ppt device on the host or the ahci device in the guest.  Maybe that is what
>     causes the failure to see the disks.
> 
>     When the guest driver attaches to the controller I see these messages on the
>     host:
>     ppt0: attempting to allocate 1 MSI vectors (1 supported)
>     ppt0: using IRQ 265 for MSI
>     and in the guest:
>     ahci0: attempting to allocate 1 MSI vectors (1 supported)
>     ahci0: using IRQ 279 for MSI
> 
>     But vmstat -i does not report any interrupts.
> 
>     Just in case, the controller has some complex topology where the AHCI PCI device
>     is behind two PCI-PCI bridges from the main bus.  But I think that that should
>     not matter and the guest needs to see only the AHCI device and the host handles
>     the bridges.
> 
>     Here are some more messages from the guest:
>     ahci0: <HighPoint RocketRAID 640 AHCI SATA controller> port
>     0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem
>     0xc000e000-0xc000efff irq 17 at device 9.0 on pci0
>     ahci0: attempting to allocate 1 MSI vectors (1 supported)
>     ahci0: using IRQ 279 for MSI
>     ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
>     ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports
>     ahcich0: <AHCI channel> at channel 0 on ahci0
>     ahcich0: Caps: CPD ESP FBSCP
>     ahcich1: <AHCI channel> at channel 1 on ahci0
>     ahcich1: Caps: CPD ESP FBSCP
>     ahcich0: AHCI reset...
>     ahcich0: SATA connect time=100us status=00000123
>     ahcich0: AHCI reset: device found
>     ahcich1: AHCI reset...
>     ahcich1: SATA connect time=100us status=00000123
>     ahcich1: AHCI reset: device found
>     ahcich1: AHCI reset: device ready after 100ms
>     ahcich1: Poll timeout on slot 1 port 15
>     ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr 00000000
>     cmd 00318017
>     ahcich0: AHCI reset: device ready after 200ms
>     ahcich0: Poll timeout on slot 1 port 15
>     ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr 00000000
>     cmd 00318017
>     And some more...
> 
>     Any ideas or suggestions?
>     Or maybe some additional information from me?
> 
>     Thank you!
> 
>     --
>     Andriy Gapon
>     _______________________________________________
>     freebsd-virtualization at freebsd.org
>     <mailto:freebsd-virtualization at freebsd.org> mailing list
>     https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>     <https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>
>     To unsubscribe, send any mail to
>     "freebsd-virtualization-unsubscribe at freebsd.org
>     <mailto:freebsd-virtualization-unsubscribe at freebsd.org>"
> 
> 


-- 
Andriy Gapon