msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation]
Harald Schmalzbauer
h.schmalzbauer at omnilan.de
Wed Oct 24 11:00:36 UTC 2012
schrieb Marius Strobl am 23.10.2012 23:12 (localtime):
> On Tue, Oct 23, 2012 at 11:49:45AM +0200, Harald Schmalzbauer wrote:
>> schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
>>> schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
>>>> Hello,
>>>>
>>>> when using igb as module, no packet is received.
>>>> If I send out anything, I see the packet with tcpdump, also the switch
>>>> learns the MAC address, but nothing comes back in - total silenc, no
>>>> boradcasts, nothing.
>>>> If I unload the module and load it again, everything works as expected!
>>>> No matter if I load it by 4th loader, or later, I always have tio unload
>>>> first then load it again.
>>>> I'ts late here, I'll see tomorrow if things change when compieled into
>>>> kernel.
>> It doesn't matter if igb is loaded as module or compiled into kernel.
>>
>>>> Maby somebody has an idea what the source of the problem could be.
>>>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and
>>>> nics are pci-passthrough.
>>> I found one possibly relevant difference:
>>>
>>> Non-Working state: dev.igb.0.link_irq: 0
>>> Working state: dev.igb.0.link_irq: 2
>> This is only true with msi-x!!!
>> If I disable mis-x, the problem itself vanishes. igb just works fine
>> from the initial loading (with dev.igb.0.link_irq=0!).
>> So dev.igb.0.link_irq is only relevant with msi-x.
>> But what makes me curious is why it also works mith mis-x enabled after
>> the second kldload!?!
> Hrm, this is strange; in r231621 (MFC'ed to stable/9 in r232092, so
> it's part of 9.1-RC2) I've blacklisted the fictitious PCI-PCI bridge
> VMware uses in case of PCI pass-through for MSI/MSI-X (as in the cases
> observed it didn't even work with a single MSI-X message). So unless
> they've changed the ID of the fictitious bridge, igb(4) should fail
> to allocate a MSI/MSI-X in the first place. Could you please provide
> the output of `pciconf -lv` on that "system"?
Hello,
here's the output:
hostb0 at pci0:0:0:0: class=0x060000 card=0x197615ad chip=0x71908086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '440BX/ZX/DX - 82443BX/ZX/DX Host bridge'
class = bridge
subclass = HOST-PCI
pcib1 at pci0:0:1:0: class=0x060400 card=0x00000000 chip=0x71918086
rev=0x01 hdr=0x01
vendor = 'Intel Corporation'
device = '440BX/ZX/DX - 82443BX/ZX/DX AGP bridge'
class = bridge
subclass = PCI-PCI
isab0 at pci0:0:7:0: class=0x060100 card=0x197615ad chip=0x71108086
rev=0x08 hdr=0x00
vendor = 'Intel Corporation'
device = '82371AB/EB/MB PIIX4 ISA'
class = bridge
subclass = PCI-ISA
atapci0 at pci0:0:7:1: class=0x01018e card=0x197615ad chip=0x71118086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82371AB/EB/MB PIIX4 IDE'
class = mass storage
subclass = ATA
intsmb0 at pci0:0:7:3: class=0x068000 card=0x197615ad chip=0x71138086
rev=0x08 hdr=0x00
vendor = 'Intel Corporation'
device = '82371AB/EB/MB PIIX4 ACPI'
class = bridge
none0 at pci0:0:7:7: class=0x088000 card=0x074015ad chip=0x074015ad
rev=0x10 hdr=0x00
vendor = 'VMware'
device = 'Virtual Machine Communication Interface'
class = base peripheral
vgapci0 at pci0:0:15:0: class=0x030000 card=0x040515ad chip=0x040515ad
rev=0x00 hdr=0x00
vendor = 'VMware'
device = 'SVGA II Adapter'
class = display
subclass = VGA
pcib2 at pci0:0:17:0: class=0x060401 card=0x079015ad chip=0x079015ad
rev=0x02 hdr=0x01
vendor = 'VMware'
device = 'PCI bridge'
class = bridge
subclass = PCI-PCI
pcib3 at pci0:0:21:0: class=0x060400 card=0x07a015ad chip=0x07a015ad
rev=0x01 hdr=0x01
vendor = 'VMware'
device = 'PCI Express Root Port'
class = bridge
subclass = PCI-PCI
pcib4 at pci0:0:22:0: class=0x060400 card=0x07a015ad chip=0x07a015ad
rev=0x01 hdr=0x01
vendor = 'VMware'
device = 'PCI Express Root Port'
class = bridge
subclass = PCI-PCI
pcib5 at pci0:0:23:0: class=0x060400 card=0x07a015ad chip=0x07a015ad
rev=0x01 hdr=0x01
vendor = 'VMware'
device = 'PCI Express Root Port'
class = bridge
subclass = PCI-PCI
pcib6 at pci0:0:24:0: class=0x060400 card=0x07a015ad chip=0x07a015ad
rev=0x01 hdr=0x01
vendor = 'VMware'
device = 'PCI Express Root Port'
class = bridge
subclass = PCI-PCI
mpt0 at pci0:3:0:0: class=0x010700 card=0x197615ad chip=0x00541000
rev=0x01 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS1068 PCI-X Fusion-MPT SAS'
class = mass storage
subclass = SAS
mps0 at pci0:4:0:0: class=0x010700 card=0x30201000 chip=0x00721000
rev=0x03 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
class = mass storage
subclass = SAS
igb0 at pci0:5:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82576 Gigabit Network Connection'
class = network
subclass = ethernet
igb1 at pci0:6:0:0: class=0x020000 card=0xa03c8086 chip=0x10c98086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82576 Gigabit Network Connection'
class = network
subclass = ethernet
0x079015ad is used for PCI-devices afaik, no matter if they're
passthrough or virtual. PCIe devices gets a 0x07a015ad slot, again no
matter if they're passtrhough or virtual.
I limited the number of functions on each slot, to have control over irq
assigning. Usually all devices would be found on pcib3, where the
virtual mpt sits. In my setup, each PCIe device gets it's own slot to
avoid forced irq sharing.
Thanks,
-Harry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20121024/ffd89098/attachment-0001.sig>
More information about the freebsd-stable
mailing list