msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation]

Harald Schmalzbauer h.schmalzbauer at omnilan.de
Tue Oct 23 09:49:55 UTC 2012


 schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
>  schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
>>  Hello,
>>
>> when using igb as module, no packet is received.
>> If I send out anything, I see the packet with tcpdump,  also the switch
>> learns the MAC address, but nothing comes back in - total silenc, no
>> boradcasts, nothing.
>> If I unload the module and load it again, everything works as expected!
>> No matter if I load it by 4th loader, or later, I always have tio unload
>> first then load it again.
>> I'ts late here, I'll see tomorrow if things change when compieled into
>> kernel.

It doesn't matter if igb is loaded as module or compiled into kernel.

>> Maby somebody has an idea what the source of the problem could be.
>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and
>> nics are pci-passthrough.
> I found one possibly relevant difference:
>
> Non-Working state:    dev.igb.0.link_irq: 0
> Working state:           dev.igb.0.link_irq: 2

This is only true with msi-x!!!
If I disable mis-x, the problem itself vanishes. igb just works fine
from the initial loading (with dev.igb.0.link_irq=0!).
So dev.igb.0.link_irq is only relevant with msi-x.
But what makes me curious is why it also works mith mis-x enabled after
the second kldload!?!

Here's the verbose comparison:

1st attaching with msi-x enabled (non-operational):
Oct 23 09:27:20 flint kernel: pci5: driver added
Oct 23 09:27:20 flint kernel: found->   vendor=0x8086, dev=0x10c9,
revid=0x01
Oct 23 09:27:20 flint kernel: domain=0, bus=5, slot=0, func=0
Oct 23 09:27:20 flint kernel: class=02-00-00, hdrtype=0x00, mfdev=0
Oct 23 09:27:20 flint kernel: cmdreg=0x0003, statreg=0x0010,
cachelnsz=16 (dwords)
Oct 23 09:27:20 flint kernel: lattimer=0x40 (1920 ns), mingnt=0x00 (0
ns), maxlat=0x00 (0 ns)
Oct 23 09:27:20 flint kernel: intpin=a, irq=16
Oct 23 09:27:20 flint kernel: powerspec 3  supports D0 D3  current D0
Oct 23 09:27:20 flint kernel: MSI supports 1 message, 64 bit, vector masks
Oct 23 09:27:20 flint kernel: MSI-X supports 10 messages in map 0x1c
Oct 23 09:27:20 flint kernel: pci0:5:0:0: reprobing on driver added
00-0x601f mem
0xd6620000-0xd663ffff,0xd6800000-0xd6bfffff,0xd6600000-0xd6603fff irq 16
at device 0.0 on pci5
Oct 23 09:27:20 flint kernel: igb0: attempting to allocate 3 MSI-X
vectors (10 supported)
Oct 23 09:27:20 flint kernel: msi: routing MSI-X IRQ 257 to local APIC 1
vector 50
Oct 23 09:27:20 flint kernel: msi: routing MSI-X IRQ 258 to local APIC 2
vector 50
Oct 23 09:27:20 flint kernel: msi: routing MSI-X IRQ 259 to local APIC 3
vector 50
Oct 23 09:27:20 flint kernel: igb0: using IRQs 257-259 for MSI-X
Oct 23 09:27:20 flint kernel: igb0: Using MSIX interrupts with 3 vectors
Oct 23 09:27:20 flint kernel: igb0: bpf attached
Oct 23 09:27:20 flint kernel: igb0: Ethernet address: 90:e2:ba:18:f8:85
Oct 23 09:27:20 flint kernel: msi: Assigning MSI-X IRQ 257 to local APIC
0 vector 51
Oct 23 09:27:20 flint kernel: igb0: Bound queue 0 to cpu 0
Oct 23 09:27:20 flint kernel: msi: Assigning MSI-X IRQ 258 to local APIC
1 vector 50
Oct 23 09:27:20 flint kernel: igb0: Bound queue 1 to cpu

2nd attaching with mis-x enabled (working):
Oct 23 09:28:12 flint kernel: pci5: driver added
Oct 23 09:28:12 flint kernel: found->   vendor=0x8086, dev=0x10c9,
revid=0x01
Oct 23 09:28:12 flint kernel: domain=0, bus=5, slot=0, func=0
Oct 23 09:28:12 flint kernel: class=02-00-00, hdrtype=0x00, mfdev=0
Oct 23 09:28:12 flint kernel: cmdreg=0x0407, statreg=0x0010,
cachelnsz=16 (dwords)
Oct 23 09:28:12 flint kernel: lattimer=0x40 (1920 ns), mingnt=0x00 (0
ns), maxlat=0x00 (0 ns)
Oct 23 09:28:12 flint kernel: intpin=a, irq=16
Oct 23 09:28:12 flint kernel: powerspec 3  supports D0 D3  current D0
Oct 23 09:28:12 flint kernel: MSI supports 1 message, 64 bit, vector masks
Oct 23 09:28:12 flint kernel: MSI-X supports 10 messages in map 0x1c
Oct 23 09:28:12 flint kernel: pci0:5:0:0: reprobing on driver added
Oct 23 09:28:12 flint kernel: igb0: <Intel(R) PRO/1000 Network
Connection version - 2.3.4> port 0x6000-0x601f mem
0xd6620000-0xd663ffff,0xd6800000-0xd6bfffff,0xd6600000-0xd6603fff irq 16
at device 0.0 on pci5
Oct 23 09:28:12 flint kernel: igb0: attempting to allocate 3 MSI-X
vectors (10 supported)
Oct 23 09:28:12 flint kernel: msi: routing MSI-X IRQ 257 to local APIC 3
vector 50
Oct 23 09:28:12 flint kernel: msi: routing MSI-X IRQ 258 to local APIC 0
vector 51
Oct 23 09:28:12 flint kernel: msi: routing MSI-X IRQ 259 to local APIC 1
vector 50
Oct 23 09:28:12 flint kernel: igb0: using IRQs 257-259 for MSI-X
Oct 23 09:28:12 flint kernel: igb0: Using MSIX interrupts with 3 vectors
Oct 23 09:28:12 flint kernel: igb0: bpf attached
Oct 23 09:28:12 flint kernel: igb0: Ethernet address: 90:e2:ba:18:f8:85
Oct 23 09:28:12 flint kernel: msi: Assigning MSI-X IRQ 257 to local APIC
0 vector 52
Oct 23 09:28:12 flint kernel: igb0: Bound queue 0 to cpu 0
Oct 23 09:28:12 flint kernel: msi: Assigning MSI-X IRQ 258 to local APIC
1 vector 51
Oct 23 09:28:12 flint kernel: igb0: Bound queue 1 to cpu 1

Since I have no idea how drivers use MSI(X), I have no idea what the
problem for this strange behaviour is.
As workarround I can disable MSI-X for the driver, or better for the
whole system, since also mps doesn't work with MSI-X enabled in my setup.
But I'd highly appreciate any help understanding what's going wrong.
It likely has something todo with my (VT-d) passthrough setup.

Thaks in advance,

-Harry



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20121023/9270b992/attachment.sig>


More information about the freebsd-stable mailing list