Possible to reset PCI device at boot? [Was: Re: msi-x enabled igb works only if module loaded twice]

[John Baldwin]

On Tuesday, October 23, 2012 8:40:44 am Harald Schmalzbauer wrote:
>  schrieb Harald Schmalzbauer am 23.10.2012 11:49 (localtime):
> >  schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
> >>  schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
> >>>  Hello,
> >>>
> >>> when using igb as module, no packet is received.
> >>> If I send out anything, I see the packet with tcpdump,  also the switch
> >>> learns the MAC address, but nothing comes back in - total silenc, no
> >>> boradcasts, nothing.
> >>> If I unload the module and load it again, everything works as expected!
> >>> No matter if I load it by 4th loader, or later, I always have tio unload
> >>> first then load it again.
> >>> I'ts late here, I'll see tomorrow if things change when compieled into
> >>> kernel.
> > It doesn't matter if igb is loaded as module or compiled into kernel.
> >
> >>> Maby somebody has an idea what the source of the problem could be.
> >>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and
> >>> nics are pci-passthrough.
> >> I found one possibly relevant difference:
> >>
> >> Non-Working state:    dev.igb.0.link_irq: 0
> >> Working state:           dev.igb.0.link_irq: 2
> > This is only true with msi-x!!!
> > If I disable mis-x, the problem itself vanishes. igb just works fine
> > from the initial loading (with dev.igb.0.link_irq=0!).
> > So dev.igb.0.link_irq is only relevant with msi-x.
> > But what makes me curious is why it also works mith mis-x enabled after
> > the second kldload!?!
>  
> I think I found the root cause:
> When ESXi powers up the guest, the passthru-devices are intialized with:
> VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 2 numVectors: 1
> intrType=2 seems to mean MSI.
> I guess, IOMMUIntel is instructed to remap one irq-vector for the
> device. But igb uses MSI-X and wants 3 vectors.
> If I unload if_igb and reload again, the ESXi-log shows the following:
> VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 4 numVectors: 3
> intrType=4 seems to mean MSI-X.
> After that initialization, if_igb works fine and saves 25kIRQ/s!
> 
> I haven't found a way to change the power-up behaviour for the guest
> with ESXi.
> 
> Is it possible to re-init a pci device from userland?

The problem is you want the igb driver to retry MSI-X even after a re-init
and that basically requires a full detach/attach, so your existing workaround
is actually the "best" way to do this. :(  Alternatively, you could try
forcing igb to not use MSI, only use either MSI-X or INTx.

-- 
John Baldwin