bhyve: Unsupported MSI-X configuration: 2/0/0

[Neel Natu]

Hi Nikos,

On Sun, Nov 30, 2014 at 7:50 AM, Nikos Vassiliadis <nvass at gmx.com> wrote:
> Hi,
>
> On 11/30/14 02:37, Neel Natu wrote:
>>
>> The "Unsupported MSI-X configuration" referred to here is that bhyve
>> doesn't emulate the 'Pending Bit Array'.
>>
>> In most cases this is not relevant because the PBA and the MSI-X
>> tables are in different page frames. In this case the MSI-X tables are
>> emulated and the pending bit array page is passed through to the
>> guest.
>>
>> If the PBA and MSI-X tables are on the same page frame then you see
>> this error. You can confirm this by doing 'pciconf -lvbc pci0:2:0:0'.
>
>
> Here is the ouput:
>>
>> bge0 at pci0:2:0:0:        class=0x020000 card=0x06471025 chip=0x16b514e4
>> rev=0x10 hdr=0x00
>>     vendor     = 'Broadcom Corporation'
>>     device     = 'NetLink BCM57785 Gigabit Ethernet PCIe'
>>     class      = network
>>     subclass   = ethernet
>>     bar   [10] = type Prefetchable Memory, range 64, base 0xb3430000, size
>> 65536, enabled
>>     bar   [18] = type Prefetchable Memory, range 64, base 0xb3440000, size
>> 65536, enabled
>>     cap 01[48] = powerspec 3  supports D0 D3  current D0
>>     cap 05[58] = MSI supports 8 messages, 64 bit enabled with 1 message
>>     cap 11[a0] = MSI-X supports 5 messages
>>                  Table in map 0x18[0x0], PBA in map 0x18[0x120]

Indeed, the MSI-X tables and PBA are both in the first page in the BAR.

>>     cap 10[ac] = PCI-Express 2 endpoint max data 128(128) link x1(x1)
>>                  speed 2.5(2.5) ASPM L1(L0s/L1)
>>     ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
>>     ecap 0003[13c] = Serial 1 0000208984c6b5ed
>>     ecap 0004[150] = Power Budgeting 1
>>     ecap 0002[160] = VC 1 max VC0
>
>
>
>
>> Having said that the drivers I have seen don't rely at all on the PBA.
>> So it may be possible that this works just fine with this patch:
>>
>> https://people.freebsd.org/~neel/patches/bhyve_ignore_unsupported_pba.patch
>>
>> If it does then I can add a tunable to bhyve to ignore this check.
>
>
> It seems a tunable can help.
>
> I tried the patch and now bhyve continues and indeed I see the bge device in
> my VM. What is interesting/strange is that the device is partly functioning.
> For example, it does detect a link change when I remove the cable but that's
> all. The kernel also complains with a "bge watchdog timeout" message.
>
> In case there is no hardware support for VT-d, as Craig noticed, how could
> this behaviour be explained? Why does link detection work?
>

I suspect that link status detection is done by polling device
registers in its MMIO space which is mapped in the guest's address
space. The MMIO accesses initiated by the guest will work independent
of VT-d.

It is only the reads/writes to system memory initiated by the device
that won't work properly without VT-d.

best
Neel

> Again, thank you both!