problem with pass-through on amd

Tue Nov 14 04:22:08 UTC 2017

Hi Andriy,
Newer AMD IOMMU has EFR capability which is missing here, something like:
ivhd0: PCI cap 0x190b640f at 0x40 feature:19<IOTLB,EFR>

Is this an old chipset? Also ivhd has fault interrupt enabled which is very
helpful in debugging:

[root at ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh
irq256: ivhd0:fault                    0          0
irq257: ivhd1:fault                    0          0

Regards,
Anish

On Mon, Nov 13, 2017 at 12:55 PM, Andriy Gapon <avg at freebsd.org> wrote:

> On 13/11/2017 22:41, Anish wrote:
> > Hi Andriy,
> >
> >>What's suspicious is that there are no interrupts either for the
> > ppt device on the host or the ahci device in the guest.
> >
> > I think most likely AMD IOMMU/amdvi is not enabled, by default it is
> disabled
> > and can be confirmed by sysctl.
> >
> > [root at ryzen /usr/home/anish]# sysctl hw.vmm.amdvi
> > hw.vmm.amdvi.domain_id: 3
> > hw.vmm.amdvi.disable_io_fault: 0
> > hw.vmm.amdvi.ptp_level: 4
> > hw.vmm.amdvi.host_ptp: 1
> > hw.vmm.amdvi.enable: 1
> > hw.vmm.amdvi.count: 2
> >
> > If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should
> work.
>
>
> I followed your how-to, so I didn't make this mistake :-)
>
> $ sysctl hw.vmm.amdvi
> hw.vmm.amdvi.domain_id: 3
> hw.vmm.amdvi.disable_io_fault: 0
> hw.vmm.amdvi.ptp_level: 4
> hw.vmm.amdvi.host_ptp: 1
> hw.vmm.amdvi.enable: 1
> hw.vmm.amdvi.count: 1
>
> Could it be that AMD-Vi support is broken in the hardware or BIOS?
> Are there any checks that I could do?
>
> Here is what I see in dmesg, but it is not a full dump of IVRS, of course:
> $ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu'
> Table 'IVRS' at 0xbdafaf80
> ACPI: IVRS 0x00000000BDAFAF80 0000F8 (v01 AMD    RD890S   00202031 AMD
> 00000000)
> AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0
> ivhd0: <AMD-Vi/IOMMU or ivhd> on acpi0
> ivhd0: Unknown dev entry:0xff
> ivhd0: Flag:3e<PassPW,ResPassPW,Isoc,IotlbSup,Coherent>
> ivhd0: max supported paging level:7 restricting to: 4
> ivhd0: device supported range [0x0 - 0xb01]
> ivhd0: device [0xa0 - 0xa0]config:d7<INIT,ExtInt,NMI,LINT0>
> ivhd0: PCI cap 0x10b540f at 0x40 feature:1<IOTLB>
> pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
>
> > On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <avg at freebsd.org
> > <mailto:avg at freebsd.org>> wrote:
> >
> >
> >     I have a FreeBSD guest on a FreeBSD host, both amd64 head.
> >     I decided to experiment with PCI pass-through and used an AHCI
> controller for
> >     the experiment.  It seems like that partially works.  The guest
> detects the
> >     controller and attaches a driver to it, the messages (reported
> capabilities,
> >     etc) look identical to those on the host.  But the guest can not
> detect any
> >     disks behind the controller.  From what I can tell, the guest has no
> problems
> >     accessing PCI configuration registers and a memory mapped BAR.  Not
> sure about
> >     the I/O ports.  What's suspicious is that there are no interrupts
> either for the
> >     ppt device on the host or the ahci device in the guest.  Maybe that
> is what
> >     causes the failure to see the disks.
> >
> >     When the guest driver attaches to the controller I see these
> messages on the
> >     host:
> >     ppt0: attempting to allocate 1 MSI vectors (1 supported)
> >     ppt0: using IRQ 265 for MSI
> >     and in the guest:
> >     ahci0: attempting to allocate 1 MSI vectors (1 supported)
> >     ahci0: using IRQ 279 for MSI
> >
> >     But vmstat -i does not report any interrupts.
> >
> >     Just in case, the controller has some complex topology where the
> AHCI PCI device
> >     is behind two PCI-PCI bridges from the main bus.  But I think that
> that should
> >     not matter and the guest needs to see only the AHCI device and the
> host handles
> >     the bridges.
> >
> >     Here are some more messages from the guest:
> >     ahci0: <HighPoint RocketRAID 640 AHCI SATA controller> port
> >     0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef
> mem
> >     0xc000e000-0xc000efff irq 17 at device 9.0 on pci0
> >     ahci0: attempting to allocate 1 MSI vectors (1 supported)
> >     ahci0: using IRQ 279 for MSI
> >     ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with
> FBS
> >     ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA
> 2ports
> >     ahcich0: <AHCI channel> at channel 0 on ahci0
> >     ahcich0: Caps: CPD ESP FBSCP
> >     ahcich1: <AHCI channel> at channel 1 on ahci0
> >     ahcich1: Caps: CPD ESP FBSCP
> >     ahcich0: AHCI reset...
> >     ahcich0: SATA connect time=100us status=00000123
> >     ahcich0: AHCI reset: device found
> >     ahcich1: AHCI reset...
> >     ahcich1: SATA connect time=100us status=00000123
> >     ahcich1: AHCI reset: device found
> >     ahcich1: AHCI reset: device ready after 100ms
> >     ahcich1: Poll timeout on slot 1 port 15
> >     ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150
> serr 00000000
> >     cmd 00318017
> >     ahcich0: AHCI reset: device ready after 200ms
> >     ahcich0: Poll timeout on slot 1 port 15
> >     ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150
> serr 00000000
> >     cmd 00318017
> >     And some more...
> >
> >     Any ideas or suggestions?
> >     Or maybe some additional information from me?
> >
> >     Thank you!
> >
> >     --
> >     Andriy Gapon
> >     _______________________________________________
> >     freebsd-virtualization at freebsd.org
> >     <mailto:freebsd-virtualization at freebsd.org> mailing list
> >     https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> >     <https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>
> >     To unsubscribe, send any mail to
> >     "freebsd-virtualization-unsubscribe at freebsd.org
> >     <mailto:freebsd-virtualization-unsubscribe at freebsd.org>"
> >
> >
>
>
> --
> Andriy Gapon
>