Possible to reset PCI device at boot? [Was: Re: msi-x enabled igb works only if module loaded twice]
John Baldwin
jhb at freebsd.org
Tue Oct 23 14:18:52 UTC 2012
On Tuesday, October 23, 2012 8:40:44 am Harald Schmalzbauer wrote:
> schrieb Harald Schmalzbauer am 23.10.2012 11:49 (localtime):
> > schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
> >> schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
> >>> Hello,
> >>>
> >>> when using igb as module, no packet is received.
> >>> If I send out anything, I see the packet with tcpdump, also the switch
> >>> learns the MAC address, but nothing comes back in - total silenc, no
> >>> boradcasts, nothing.
> >>> If I unload the module and load it again, everything works as expected!
> >>> No matter if I load it by 4th loader, or later, I always have tio unload
> >>> first then load it again.
> >>> I'ts late here, I'll see tomorrow if things change when compieled into
> >>> kernel.
> > It doesn't matter if igb is loaded as module or compiled into kernel.
> >
> >>> Maby somebody has an idea what the source of the problem could be.
> >>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and
> >>> nics are pci-passthrough.
> >> I found one possibly relevant difference:
> >>
> >> Non-Working state: dev.igb.0.link_irq: 0
> >> Working state: dev.igb.0.link_irq: 2
> > This is only true with msi-x!!!
> > If I disable mis-x, the problem itself vanishes. igb just works fine
> > from the initial loading (with dev.igb.0.link_irq=0!).
> > So dev.igb.0.link_irq is only relevant with msi-x.
> > But what makes me curious is why it also works mith mis-x enabled after
> > the second kldload!?!
>
> I think I found the root cause:
> When ESXi powers up the guest, the passthru-devices are intialized with:
> VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 2 numVectors: 1
> intrType=2 seems to mean MSI.
> I guess, IOMMUIntel is instructed to remap one irq-vector for the
> device. But igb uses MSI-X and wants 3 vectors.
> If I unload if_igb and reload again, the ESXi-log shows the following:
> VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 4 numVectors: 3
> intrType=4 seems to mean MSI-X.
> After that initialization, if_igb works fine and saves 25kIRQ/s!
>
> I haven't found a way to change the power-up behaviour for the guest
> with ESXi.
>
> Is it possible to re-init a pci device from userland?
The problem is you want the igb driver to retry MSI-X even after a re-init
and that basically requires a full detach/attach, so your existing workaround
is actually the "best" way to do this. :( Alternatively, you could try
forcing igb to not use MSI, only use either MSI-X or INTx.
--
John Baldwin
More information about the freebsd-stable
mailing list