[PATCH] Support PCIe Alternative RID Interpretation (ARI)

[Neel Natu]

Hi Ryan,

On Tue, Mar 18, 2014 at 10:51 AM, Ryan Stone <rysto32 at gmail.com> wrote:
> On Mon, Mar 17, 2014 at 5:20 PM, Neel Natu <neelnatu at gmail.com> wrote:
>> Hi Ryan,
>> In any case it seems that the VT-d implementation in bhyve will work
>> fine with ARI enabled devices.
>
> There was an assert that would trip for the function number being
> greater than 8.
>
>
> I've put together the following patch series:
>
> http://people.freebsd.org/~rstone/patches/ari/0001-Add-a-method-to-get-the-PCI-RID-for-a-device.patch
>
> This adds a method to get the RID that will be consumed by the VT-d
> drivers.  This patch is non-ARI only.
>
>
> http://people.freebsd.org/~rstone/patches/ari/0002-Re-implement-the-DMAR-I-O-MMU-code-in-terms-of-PCI-R.patch
>
> This reworks the busdma DMAR code to work in terms of RIDs where
> necessary.  This should be a no-op.  I tested this with
> hw.dmar.enable=1 on a Nehalem with the em driver and a Sandy Bridge
> with the igb and ixgbe driver and was able to pass traffic.
>
>
> http://people.freebsd.org/~rstone/patches/ari/0003-Re-write-bhyve-s-I-O-MMU-handling-in-terms-of-PCI-RI.patch
>
> Same thing, but for bhyve.  I'm not sure that passing the rid down to
> the CPU-dependent layer is the right thing to do, because I'm not sure
> what the AMD VT-d equivalent requires.  Should I just pass down the
> entire device_t and let the CPU layer deal with it?  I tested loading
> vmm.ko with a device assigned to the ppt driver but I didn't go as far
> as starting a VM and using PCI passthrough.
>

The bhyve portion of the patch looks fine to me.

A cursory read of the AMD IOMMU spec suggests that the translation
table is indexed using the 16-bit RID so passing it into the
CPU-dependent layer will work fine.

best
Neel

> (Also, as you'd probably expected doing this with hw.dmar.enable=1
> causes all hell to break loose).
>
>
> http://people.freebsd.org/~rstone/patches/ari/0004-Add-support-for-PCIe-ARI.patch
>
> This is a slightly reworked version of the previous patch.  The main
> difference are that there is a new implementation of pcib_get_rid that
> understands ARI RIDs.  I also fixed a bug where the default
> implementation of pcib_numslots was not actually being used because I
> misspelled DEFAULT as default in pcib_if.m.
>
>
> http://people.freebsd.org/~rstone/patches/ari/0005-Print-status-of-ARI-capability-in-pciconf-c.patch
>
> This makes pciconf -c dump the status of ARI on PCIe downstream ports.