msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation]

[Marius Strobl]

On Tue, Oct 23, 2012 at 11:49:45AM +0200, Harald Schmalzbauer wrote:
>  schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
> >  schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
> >>  Hello,
> >>
> >> when using igb as module, no packet is received.
> >> If I send out anything, I see the packet with tcpdump,  also the switch
> >> learns the MAC address, but nothing comes back in - total silenc, no
> >> boradcasts, nothing.
> >> If I unload the module and load it again, everything works as expected!
> >> No matter if I load it by 4th loader, or later, I always have tio unload
> >> first then load it again.
> >> I'ts late here, I'll see tomorrow if things change when compieled into
> >> kernel.
> 
> It doesn't matter if igb is loaded as module or compiled into kernel.
> 
> >> Maby somebody has an idea what the source of the problem could be.
> >> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and
> >> nics are pci-passthrough.
> > I found one possibly relevant difference:
> >
> > Non-Working state:    dev.igb.0.link_irq: 0
> > Working state:           dev.igb.0.link_irq: 2
> 
> This is only true with msi-x!!!
> If I disable mis-x, the problem itself vanishes. igb just works fine
> from the initial loading (with dev.igb.0.link_irq=0!).
> So dev.igb.0.link_irq is only relevant with msi-x.
> But what makes me curious is why it also works mith mis-x enabled after
> the second kldload!?!

Hrm, this is strange; in r231621 (MFC'ed to stable/9 in r232092, so
it's part of 9.1-RC2) I've blacklisted the fictitious PCI-PCI bridge
VMware uses in case of PCI pass-through for MSI/MSI-X (as in the cases
observed it didn't even work with a single MSI-X message). So unless
they've changed the ID of the fictitious bridge, igb(4) should fail
to allocate a MSI/MSI-X in the first place. Could you please provide
the output of `pciconf -lv` on that "system"?

Marius