em(4) duplex problems with 82541EI on RELENG_8, -CURRENT on PowerEdge 1850

[Jack Vogel]

Yes, the commits today are slated to get into 8.1, at least that's my
understanding.

Jack


On Fri, Jun 18, 2010 at 10:46 AM, Brandon Gooch <jamesbrandongooch at gmail.com
> wrote:

> On Tue, Jun 1, 2010 at 2:37 PM, Jeremy Chadwick
> <freebsd at jdc.parodius.com> wrote:
> > On Tue, Jun 01, 2010 at 03:18:39PM -0400, Brian A. Seklecki wrote:
> >> = Re-posted from freebsd-hardware@, since this is more of a bug
> >>  report than a hardware comparability inquiry / buying strategy
> >>  discussion. ==
> >>
> >> All:
> >>
> >>   Has anyone upgraded their PowerEdge 1850s to 8.0-PL or
> >>   RELENG_8 -stable?  We're seeing problems where 7.2-PL and
> >>   6.3-PL were not affected on the same hardware.
> >>
> >>   The problem is that forcing the duplex 100/full on both
> >>   sides no longer functions.
> >>
> >>   Configuration:
> >>
> >>    - A variety of Cisco L2/L3 switches over the last decade:
> >>    -- 2848G-L3
> >>    -- 2950
> >>    -- 2960s
> >>    -- 3550-12Ts
> >>    -- 3550XLs
> >>    -- Duplex forced 100/full on Cisco side
> >>    - FreeBSD/amd64 RELENG_8 or 9-CURRENT with duplex
> >>      forced '100baseTX mediaopt full-duplex',
> >>    - This configuration has worked since FreeBSD 5.4
> >>
> >>   When connected to PowerEdge 1850r1/r2, with the onboard Intel
> >>   82541EI, the parenthesis show an actual media speed/duplex of:
> >>
> >>   media: Ethernet 100baseTX <full-duplex>  (100baseTX <half-duplex>)
> >>
> >>   The same configuration using a Dell-sold Intel dual port
> >>   82546EB, in the same system, on the same switch, works fine.
> >>
> >>
> >> -----------------
> >> ifconfig(8):
> >> -----------------
> >> em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX, \r
> >>                  MULTICAST> metric 0 mtu 1500
> >> options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
> >> ether 00:13:72:4f:70:81
> >> inet 192.168.97.20 netmask 0xffffff80 broadcast 192.168.97.127
> >> media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
> >> status: active
> >> -----------------
> >> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,\
> >>                  MULTICAST> metric 0 mtu 1500>
> >> options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
> >> ether 00:04:23:c8:fe:ac
> >> media: Ethernet 100baseTX <full-duplex>
> >> status: active
> >> -----------------
> >> -----------------
> >> pciconf(8):
> >> -----------------
> >> em3 at pci0:7:8:0:       class=0x020000 card=0x016d1028 chip=0x10768086
> >>                 rev=0x05 hdr=0x00
> >>     vendor     = 'Intel Corporation'
> >>     device     = 'Gigabit Ethernet Controller (82541EI)'
> >>     class      = network
> >>     subclass   = ethernet
> >> em0 at pci0:3:11:0:      class=0x020000 card=0x10128086 chip=0x10108086
> rev=0x01
> >> hdr=0x00
> >>    vendor     = 'Intel Corporation'
> >>    device  = 'Dual Port Gigabit Ethernet Controller (Copper) (82546EB)'
> >>    class      = network
> >>    subclass   = ethernet
> >>
> >> -----------------
> >>
> >> rc.conf(5) for shits & giggles:
> >>
> >> ifconfig_em0="inet X netmask Y media 100baseTX mediaopt full-duplex"
> >> ifconfig_em3="inet Z netmask F media 100baseTX mediaopt full-duplex"
> >>
> >> --------
> >>
> >> Example IOS switch config:
> >>  interface FastEthernet0/39
> >>   description I hate Dell
> >>   switchport access vlan 100
> >>   switchport mode access
> >>   speed 100
> >>   duplex full
> >>   spanning-tree portfast
> >>  end
> >> --------
> >>
> >> I've been clearing interface counters on the switch side, but I'll send
> >> 'netstat -i', 'show interface counters', and 'sudo sysctl -w
> >> dev.em.3.stats=1' ASAP to illustrate connectivity errors soon.
> >>
> >> Are we being punished for patronizing Dell?
> >>
> >> Is it possible that ifconfig(8) output has simply changed?  Are the
> >> values in the parenthesis on the right the Ethernet auto-sense desired
> >> values where as outside the parenthesis the current active values?
> >>
> >> In 6.3/7.2, once you forced a speed/duplex, the values in parenthesis
> >> went away entirely.
> >>
> >> The only way I've been able to make that happen is to #define in
> >> src/sys/dev/e1000/if_em.h:
> >>
> >>   #define DO_AUTO_NEG 0
> >>   /*
> >>    * This parameter control whether or not the driver will wait for
> >>    * autonegotiation to complete.
> >>    * 1 - Wait for autonegotiation to complete
> >>    * 0 - Don't wait for autonegotiation to complete
> >>   */
> >>
> >> Also seems odd that some ICs are affected but not others.
> >>
> >> Its also possible that my problems are pf(4) + setfib(8) related and I
> >> that this is a separate issue.
> >>
> >> Two new notes since the original post:
> >>
> >>  - I have confirmed this problem on two revisions of the Dell
> >>    8th gen hardware in two different datacenters
> >>  - The problem persists on -CURRENT from 05/2010
> >>  - RELENG_7 does not seem to be impacted
> >>  - More stats below.
> >>
> >>
> >> Thanks,
> >>     ~BAS
> >>
> >> ---------------
> >>
> >>
> >>
> >> em1: link state changed to DOWN
> >> em1: link state changed to UP
> >> em1: link state changed to DOWN
> >> em1: link state changed to UP
> >> em1: link state changed to DOWN
> >> em1: link state changed to UP
> >> em1: link state changed to DOWN
> >> em1: link state changed to UP
> >> em1: link state changed to DOWN
> >> em1: link state changed to UP
> >> em1: link state changed to DOWN
> >>
> >> em0: Excessive collisions = 0
> >> em0: Sequence errors = 0
> >> em0: Defer count = 0
> >> em0: Missed Packets = 0
> >> em0: Receive No Buffers = 0
> >> em0: Receive Length Errors = 0
> >> em0: Receive errors = 0
> >> em0: Crc errors = 0
> >> em0: Alignment errors = 0
> >> em0: Collision/Carrier extension errors = 0
> >> em0: RX overruns = 0
> >> em0: watchdog timeouts = 0
> >> em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
> >> em0: XON Rcvd = 0
> >> em0: XON Xmtd = 0
> >> em0: XOFF Rcvd = 0
> >> em0: XOFF Xmtd = 0
> >> em0: Good Packets Rcvd = 1319916
> >> em0: Good Packets Xmtd = 1070646
> >> em0: TSO Contexts Xmtd = 0
> >> em0: TSO Contexts Failed = 0
> >> em1: Excessive collisions = 0
> >> em1: Sequence errors = 0
> >> em1: Defer count = 0
> >> em1: Missed Packets = 0
> >> em1: Receive No Buffers = 0
> >> em1: Receive Length Errors = 0
> >> em1: Receive errors = 0
> >> em1: Crc errors = 0
> >> em1: Alignment errors = 0
> >> em1: Collision/Carrier extension errors = 0
> >> em1: RX overruns = 0
> >> em1: watchdog timeouts = 0
> >> em1: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
> >> em1: XON Rcvd = 0
> >> em1: XON Xmtd = 0
> >> em1: XOFF Rcvd = 0
> >> em1: XOFF Xmtd = 0
> >> em1: Good Packets Rcvd = 251348
> >> em1: Good Packets Xmtd = 204160
> >> em1: TSO Contexts Xmtd = 0
> >> em1: TSO Contexts Failed = 0
> >>
> >> --------------------
> >>
> >>
> >> as0# sh int fa0/43
> >> FastEthernet0/43 is up, line protocol is up (connected)
> >> Hardware is Fast Ethernet, address is 0015.c683.51ab (bia
> >> 0015.c683.51ab)
> >> Description: X-Server EM1
> >> MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
> >> reliability 255/255, txload 1/255, rxload 1/255
> >> Encapsulation ARPA, loopback not set
> >> Keepalive set (10 sec)
> >> Full-duplex, 100Mb/s, media type is 100BaseTX
> >> input flow-control is unsupported output flow-control is unsupported
> >> ARP type: ARPA, ARP Timeout 04:00:00
> >> Last input never, output 00:00:08, output hang never
> >> Last clearing of "show interface" counters 6d03h
> >> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> >> Queueing strategy: fifo
> >> Output queue: 0/40 (size/max)
> >> 5 minute input rate 0 bits/sec, 0 packets/sec
> >> 5 minute output rate 1000 bits/sec, 3 packets/sec
> >> 291422 packets input, 131521274 bytes, 0 no buffer
> >> Received 798 broadcasts (0 multicast)
> >> 0 runts, 0 giants, 0 throttles
> >> 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
> >> 0 watchdog, 99 multicast, 0 pause input
> >> 0 input packets with dribble condition detected
> >> 651929 packets output, 73550092 bytes, 0 underruns
> >> 0 output errors, 0 collisions, 4 interface resets
> >> 0 babbles, 0 late collision, 0 deferred
> >> 0 lost carrier, 0 no carrier, 0 PAUSE output
> >> 0 output buffer failures, 0 output buffers swapped out
> >
> > Brian, could you please provide the following output?
> >
> > - uname -a  (you can X-out the machine name if need be)
> > - dmesg | egrep 'em0|em3'  (provides em driver version number)
> > - pciconf -lvc  (this will differ from what you provided above)
> >
> > Next, some of the stats you provided are for em1 when most of your post
> > focuses around em0 and em3.  Is there some correlation or was it a
> > mistake?
> >
> > Adding Jack Vogel of Intel to the CC list, as he's been working on em(4)
> > as of late.
>
> Brian, I have no idea if this will help or not, but...
>
> Jack just committed bits to the Intel drivers (em(4) ixgbe(4)), will
> you have a chance to test a new build? I'm trying to find an unused
> system ATM to test on myself, but it may take me a day or to.
>
> BTW, it appears Jack may be trying to get the fixes (and features)
> into 8.1-RELEASE, let's hope that it happens :)
>
> -Brandon
>