Possible to reset PCI device at boot? [Was: Re: msi-x enabled igb works only if module loaded twice]

[Harald Schmalzbauer]

 schrieb Harald Schmalzbauer am 23.10.2012 11:49 (localtime):
>  schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
>>  schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
>>>  Hello,
>>>
>>> when using igb as module, no packet is received.
>>> If I send out anything, I see the packet with tcpdump,  also the switch
>>> learns the MAC address, but nothing comes back in - total silenc, no
>>> boradcasts, nothing.
>>> If I unload the module and load it again, everything works as expected!
>>> No matter if I load it by 4th loader, or later, I always have tio unload
>>> first then load it again.
>>> I'ts late here, I'll see tomorrow if things change when compieled into
>>> kernel.
> It doesn't matter if igb is loaded as module or compiled into kernel.
>
>>> Maby somebody has an idea what the source of the problem could be.
>>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and
>>> nics are pci-passthrough.
>> I found one possibly relevant difference:
>>
>> Non-Working state:    dev.igb.0.link_irq: 0
>> Working state:           dev.igb.0.link_irq: 2
> This is only true with msi-x!!!
> If I disable mis-x, the problem itself vanishes. igb just works fine
> from the initial loading (with dev.igb.0.link_irq=0!).
> So dev.igb.0.link_irq is only relevant with msi-x.
> But what makes me curious is why it also works mith mis-x enabled after
> the second kldload!?!
 
I think I found the root cause:
When ESXi powers up the guest, the passthru-devices are intialized with:
VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 2 numVectors: 1
intrType=2 seems to mean MSI.
I guess, IOMMUIntel is instructed to remap one irq-vector for the
device. But igb uses MSI-X and wants 3 vectors.
If I unload if_igb and reload again, the ESXi-log shows the following:
VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 4 numVectors: 3
intrType=4 seems to mean MSI-X.
After that initialization, if_igb works fine and saves 25kIRQ/s!

I haven't found a way to change the power-up behaviour for the guest
with ESXi.

Is it possible to re-init a pci device from userland?

Thanks,

-Harry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20121023/54479bff/attachment.sig>