problem with pass-through on amd

[Andriy Gapon]

On 14/11/2017 06:22, Anish wrote:
> [root at ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh
> irq256: ivhd0:fault                    0          0
> irq257: ivhd1:fault                    0          0

After I fixed the MSI setup problem that I described in the previous email I was
finally able to see the real problem.
Now I have:

irq263: ivhd0:fault                    9          0

dev.ivhd.0.event_tail: 240
dev.ivhd.0.event_head: 240
dev.ivhd.0.event_intr_count: 9


I've got a bunch of messages like these in the log:
[IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x70f70400x30<PR,RW>]
[IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x2fdf0400x30<PR,RW>]
(BTW, there seems to be a missing space before 0x30)

Now it's obvious what the problem is.
The controller has a RID of 0xa00 (its PCI locator is 10:0:0) but the requests
are coming from 0xa01.  The card actually has another PCI device at that
location (10:0:1), it's a vestigial IDE controller (in a sense that it is not
connected to any ports, so it cannot really provide any functionality).

I've passed both of the PCI device to bhyve and everything started working.
Thanks!

I've googled a little bit and it seems that this is not an uncommon problem.
Linux has a bunch of quirks for this and similar kinds of problems (they call it
PCI or DMA aliasing):
http://elixir.free-electrons.com/linux/v4.8.16/source/drivers/pci/quirks.c#L3701
The comments say that sometimes requests come from devices that do not exist at
all (or hidden beyond non-transparent bridges).

An interesting new world for me :-)

By the way, my understanding is that the alias IVHD entries in IVRS are designed
to report such aliasing issues when they are known to the platform.
But it appears to me that FreeBSD currently ignores those aliases.
Also, it looks like we are not making any use of IVHD entries beyond printing
them.  Based on the specification I think that we should use IVHD flags to set
up appropriate bits in the corresponding device table entries.  But we are not
doing that.  That's probably okay since we do not support the pass-though for
devices with complex properties anyway.

-- 
Andriy Gapon