msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation]

[Jack Vogel]

LOL, wow this is interesting. When I first was developing the VF support,
and using
Linux KVM as the host I had exactly this problem, turned out it was because
of the
way in which MSIX was handled in the Linux side. When my driver would first
attempt
to get some vectors the Linux code would go look at the vector table, but
the way
the PCI code in FreeBSD works that table is not set up yet, so Linux would
see no
legitimate vectors in the table, and decide the guest was ineligible for
MSIX :(, but
by this time the FreeBSD code actually DID write some vectors into the
table, and
thus when you load the driver the SECOND time Linux would see the table
populated
and TADA!! would enable the guest to use MSIX.

Now, I went thru a bunch of efforts via our Linux team here to have the KVM
code fixed,
its design was bogus, and I believe it has been, but it sounds like maybe
VMWare has
the same broken design??

There is nothing you can do about this because the issue is in the host,
not the guest,
well getting the host code fixed is the solution :)

Hope this helps,

Jack


On Tue, Oct 23, 2012 at 2:12 PM, Marius Strobl <marius at alchemy.franken.de>wrote:

> On Tue, Oct 23, 2012 at 11:49:45AM +0200, Harald Schmalzbauer wrote:
> >  schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
> > >  schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
> > >>  Hello,
> > >>
> > >> when using igb as module, no packet is received.
> > >> If I send out anything, I see the packet with tcpdump,  also the
> switch
> > >> learns the MAC address, but nothing comes back in - total silenc, no
> > >> boradcasts, nothing.
> > >> If I unload the module and load it again, everything works as
> expected!
> > >> No matter if I load it by 4th loader, or later, I always have tio
> unload
> > >> first then load it again.
> > >> I'ts late here, I'll see tomorrow if things change when compieled into
> > >> kernel.
> >
> > It doesn't matter if igb is loaded as module or compiled into kernel.
> >
> > >> Maby somebody has an idea what the source of the problem could be.
> > >> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and
> > >> nics are pci-passthrough.
> > > I found one possibly relevant difference:
> > >
> > > Non-Working state:    dev.igb.0.link_irq: 0
> > > Working state:           dev.igb.0.link_irq: 2
> >
> > This is only true with msi-x!!!
> > If I disable mis-x, the problem itself vanishes. igb just works fine
> > from the initial loading (with dev.igb.0.link_irq=0!).
> > So dev.igb.0.link_irq is only relevant with msi-x.
> > But what makes me curious is why it also works mith mis-x enabled after
> > the second kldload!?!
>
> Hrm, this is strange; in r231621 (MFC'ed to stable/9 in r232092, so
> it's part of 9.1-RC2) I've blacklisted the fictitious PCI-PCI bridge
> VMware uses in case of PCI pass-through for MSI/MSI-X (as in the cases
> observed it didn't even work with a single MSI-X message). So unless
> they've changed the ID of the fictitious bridge, igb(4) should fail
> to allocate a MSI/MSI-X in the first place. Could you please provide
> the output of `pciconf -lv` on that "system"?
>
> Marius
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>