bug with some em nics on RELENG_7
Jack Vogel
jfvogel at gmail.com
Thu Nov 19 00:29:19 UTC 2009
Hey Mike,
Can you check if you see the same behavior on RELENG 8?
There is a systemic problem having to do with when to enable interrupts that
might be behind this. The em driver does not enable them until
em_init_locked(),
this is because until then its not ready to deal with a TX or RX interrupt.
However,
this means that a Link interrupt also will not be seen, BUT, and here is
where it
gets a bit funny, an call to check link happens in attach, it will be either
true or
false, AND, even if you remove or add a cable after that point, until
interrupts
are enabled the state will not change.
In the days before MSIX one interrupt was for everything so it was
impossible
to change this without a radical rework to the driver design, but I suppose
it
would be possible with MSIX to selectively enable the link one earlier, I
seem
to recall discussions with our Linux crew that made me decide not to pursue
that (its of limited value really).
Not sure why this happens on Hartwell (82574) and not on 82571, that's
an interesting bit, the 82574 is the ONLY interface in the em driver that
has MSIX support, unfortunately its kinda hacked in, but it did not really
fit into the igb driver either for various technical reasons.
What if you boot up, then do NOT ping or anything until the interface is
assigned an address (and so init is run), and the cable is plugged in. If
that happens first does it work?
Do let me know if you can check on 8, if not I can have my validation
engineer try this.
Best regards,
Jack
On Wed, Nov 18, 2009 at 1:30 PM, Mike Tancsa <mike at sentex.net> wrote:
>
> On two Intel chipset Supermicro boards (X8STi and X8STE-0) using the
> onboard em nics (dmesg info below), I seem to have run into an issue where
> if I boot the box up with the cables unplugged, I cannot get the NICS to
> properly work post boot up. This is quite repeatable for me. So at boot
> time, I have
>
> # ifconfig em5
> em5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
> ether 00:30:48:d6:ef:13
> inet 3.3.3.3 netmask 0xffffff00 broadcast 3.3.3.255
> media: Ethernet autoselect
> status: no carrier
>
>
> I then ping something that would be across the wire while the nic is down.
> e.g. ping 3.3.3.1
>
> I then plug in the cable so that the other side has 3.3.3.1
>
> ifconfig shows all looks good
>
> # ifconfig em5
> em5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
> ether 00:30:48:d6:ef:13
> inet 3.3.3.3 netmask 0xffffff00 broadcast 3.3.3.255
> media: Ethernet autoselect (1000baseTX <full-duplex>)
> status: active
>
> I try and ping 3.3.3.1 which is on xover (via a switch shows the same
> behaviour), and no response to the pings.... BUT, I do see the MAC addr show
> up
> # ping -c 2 -S 3.3.3.3 3.3.3.1
> PING 3.3.3.1 (3.3.3.1) from 3.3.3.3: 56 data bytes
>
> --- 3.3.3.1 ping statistics ---
> 2 packets transmitted, 0 packets received, 100.0% packet loss
> # arp -na
> ? (3.3.3.1) at 00:30:48:94:88:20 on em5 [ethernet]
> ? (3.3.3.3) at 00:30:48:d6:ef:13 on em5 permanent [ethernet]
>
> I can see its mac addr ?!?
>
> Furthermore, if I do
>
> # ifconfig em5 3.3.3.55/32 alias
>
> On the other side, I see
>
> 0(ich10)# tcpdump -nei igb0
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on igb0, link-type EN10MB (Ethernet), capture size 96 bytes
> 16:16:03.380886 00:30:48:d6:ef:13 > ff:ff:ff:ff:ff:ff, ethertype ARP
> (0x0806), length 60: Request who-has 3.3.3.55 tell 3.3.3.55, length 46
>
>
> and I can ping if I specify the alias as the source IP
>
> # ping -S 3.3.3.55 3.3.3.1
> PING 3.3.3.1 (3.3.3.1) from 3.3.3.55: 56 data bytes
> 64 bytes from 3.3.3.1: icmp_seq=0 ttl=64 time=0.184 ms
> 64 bytes from 3.3.3.1: icmp_seq=1 ttl=64 time=0.051 ms
> 64 bytes from 3.3.3.1: icmp_seq=2 ttl=64 time=0.055 ms
>
>
>
> 16:17:01.603345 00:30:48:d6:ef:13 > 00:30:48:94:88:20, ethertype ARP
> (0x0806), length 60: Reply 3.3.3.55 is-at 00:30:48:d6:ef:13, length 46
> 16:17:01.603349 00:30:48:94:88:20 > 00:30:48:d6:ef:13, ethertype IPv4
> (0x0800), length 98: 3.3.3.1 > 3.3.3.55: ICMP echo reply, id 7946, seq 0,
> length 64
> 16:17:02.603497 00:30:48:d6:ef:13 > 00:30:48:94:88:20, ethertype IPv4
> (0x0800), length 98: 3.3.3.55 > 3.3.3.1: ICMP echo request, id 7946, seq
> 1, length 64
> 16:17:02.603502 00:30:48:94:88:20 > 00:30:48:d6:ef:13, ethertype IPv4
> (0x0800), length 98: 3.3.3.1 > 3.3.3.55: ICMP echo reply, id 7946, seq 1,
> length 64
> 16:17:03.604510 00:30:48:d6:ef:13 > 00:30:48:94:88:20, ethertype IPv4
> (0x0800), length 98: 3.3.3.55 > 3.3.3.1: ICMP echo request, id 7946, seq
> 2, length 64
> 16:17:03.604516 00:30:48:94:88:20 > 00:30:48:d6:ef:13, ethertype IPv4
> (0x0800), length 98: 3.3.3.1 > 3.3.3.55: ICMP echo reply, id 7946, seq 2,
> length 64
>
>
>
> but not using the initial IP addr
>
> 0[iolite3A]# ping -S 3.3.3.3 3.3.3.1
> PING 3.3.3.1 (3.3.3.1) from 3.3.3.3: 56 data bytes
> ^C
> --- 3.3.3.1 ping statistics ---
> 2 packets transmitted, 0 packets received, 100.0% packet loss
> #
>
> Yet,
>
> # ping -S 3.3.3.3 3.3.3.1
> PING 3.3.3.1 (3.3.3.1) from 3.3.3.3: 56 data bytes
> ^C
> --- 3.3.3.1 ping statistics ---
> 2 packets transmitted, 0 packets received, 100.0% packet loss
> # ping -S 3.3.3.4 3.3.3.1
> PING 3.3.3.1 (3.3.3.1) from 3.3.3.4: 56 data bytes
> 64 bytes from 3.3.3.1: icmp_seq=0 ttl=64 time=0.176 ms
> 64 bytes from 3.3.3.1: icmp_seq=1 ttl=64 time=0.050 ms
> ^C
> --- 3.3.3.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.050/0.113/0.176/0.063 ms
>
>
> Strange, eh ?
>
>
> em4 at pci0:6:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 rev=0x00
> hdr=0x00
> vendor = 'Intel Corporation'
> class = network
> subclass = ethernet
> cap 01[c8] = powerspec 2 supports D0 D3 current D0
> cap 05[d0] = MSI supports 1 message, 64 bit
> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
> em5 at pci0:7:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 rev=0x00
> hdr=0x00
> vendor = 'Intel Corporation'
> class = network
> subclass = ethernet
> cap 01[c8] = powerspec 2 supports D0 D3 current D0
> cap 05[d0] = MSI supports 1 message, 64 bit
> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
>
>
> em4: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xdc00-0xdc1f mem
> 0xfaee0000-0xfaefffff,0xfaedc000-0xfaedffff irq 16 at device 0.0 on pci6
> em4: Using MSIX interrupts
> em4: [ITHREAD]
> em4: [ITHREAD]
> em4: [ITHREAD]
> em4: Ethernet address: 00:30:48:d6:ef:12
> pcib7: <ACPI PCI-PCI bridge> irq 16 at device 28.1 on pci0
> pci7: <ACPI PCI bus> on pcib7
> em5: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xec00-0xec1f mem
> 0xfafe0000-0xfaffffff,0xfafdc000-0xfafdffff irq 17 at device 0.0 on pci7
> em5: Using MSIX interrupts
> em5: [ITHREAD]
> em5: [ITHREAD]
> em5: [ITHREAD]
> em5: Ethernet address: 00:30:48:d6:ef:13
>
> The same does NOT happen with my external 2 port pcie nics (it says HP, but
> they are intel branded)
> eg
> em0 at pci0:1:0:0: class=0x020000 card=0x115e8086 chip=0x105e8086 rev=0x06
> hdr=0x00
> vendor = 'Intel Corporation'
> device = 'HP NC360T PCIe DP Gigabit Server Adapter (n1e5132)'
> class = network
> subclass = ethernet
> cap 01[c8] = powerspec 2 supports D0 D3 current D0
> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x4(x4)
> em1 at pci0:1:0:1: class=0x020000 card=0x115e8086 chip=0x105e8086 rev=0x06
> hdr=0x00
> vendor = 'Intel Corporation'
> device = 'HP NC360T PCIe DP Gigabit Server Adapter (n1e5132)'
> class = network
> subclass = ethernet
> cap 01[c8] = powerspec 2 supports D0 D3 current D0
> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x4(x4)
>
> ---Mike
>
> --------------------------------------------------------------------
> Mike Tancsa, tel +1 519 651 3400
> Sentex Communications, mike at sentex.net
> Providing Internet since 1994 www.sentex.net
> Cambridge, Ontario Canada www.sentex.net/mike
>
>
More information about the freebsd-stable
mailing list