NFE adapter 'hangs'

[Pyun YongHyeon]

On Fri, Sep 03, 2010 at 07:59:26AM +0100, Melissa Jenkins wrote:
> 
> Thank you for your very quick response :)
> 

[...]

> >Also I'd like to know whether both RX and TX are dead or only one
> >RX/TX path is hung. Can you see incoming traffic with tcpdump when
> >you think the controller is in stuck?
> 
> Yes, though not very much. The traffic to 4800 is every second so you can see in the following trace when it stops
> 
> 07:10:42.287163 IP 192.168.1.203 > 224.0.0.240:  pfsync 108
> 07:10:42.911995
> 07:10:43.112073 STP 802.1d, Config, Flags [Topology change], bridge-id 8000.c4:7d:4f:a9:ac:30.8008, length 43
> 07:10:43.148659 IP 192.168.1.203.57026 > 192.168.1.255.4800: UDP, length 60
> 07:10:43.148684 IP 172.31.1.203 > 172.31.1.129: GREv0, length 92: IP 192.168.1.203.57026 > 192.168.1.129.4800: UDP, length 60
> 07:10:43.148689 IP 172.31.1.203 > 172.31.1.129: GREv0, length 92: IP 192.168.1.203.57026 > 192.168.1.1.4800: UDP, length 60
> 07:10:43.148918 IP 192.168.1.213.40677 > 192.168.1.255.4800: UDP, length 48

[...]

> a bit later on, still broken, a slight odd message:
> 07:11:43.079720 IP 172.31.1.129 > 172.31.1.213: GREv0, length 52: IP 192.168.1.129.60446 > 192.168.1.213.179:  tcp 12 [bad hdr length 16 - too short, < 20]
> 07:11:44.210794 IP 172.31.1.129 > 172.31.1.203: GREv0, length 84: IP 192.168.1.129.64744 > 192.168.1.203.4800: UDP, length 52
> 07:11:44.210831 IP 172.31.1.129 > 172.31.1.213: GREv0, length 84: IP 192.168.1.129.64744 > 192.168.1.213.4800: UDP, length 52
> 
> Now this really is odd, I don't recognise either of those MAC addresses, though the SQL shown is used on this machine (
> 07:12:13.054393 45:43:54:20:41:63 > 00:00:03:53:45:4c, ethertype Unknown (0x6374), length 60:
>         0x0000:  556e 6971 7565 4964 2046 524f 4d20 7261  UniqueId.FROM.ra
>         0x0010:  6461 6363 7420 2057 4845 5245 2043 616c  dacct..WHERE.Cal
>         0x0020:  6c69 6e67 5374 6174 696f 6e49 6420       lingStationId.

Hmm, it seems you're using really complex setup. It's very hard to
narrow down guilty ones under these environments. Could you setup
simple network configuration that reproduces the issue? One of
possible cause would be wrong(garbled) data might be passed up to
upper stack. But I have no idea why you see GRE packets with
truncated TCP header(172.31.1.129 > 172.31.1.213).
How about disabling TX/RX checksum offloading as well as TSO?

[...]

> 
> I then restarted the interface (nfe down/up, route restart)
> 
> From dmesg at the time (slight obfuscated)
> Sep  3 07:10:19 manch2 bgpd[89612]: neighbor XX: received notification: HoldTimer expired, unknown subcode 0
> Sep  3 07:10:49 manch2 bgpd[89612]: neighbor XX connect: Host is down
> # at this point I took the interface down & up and reloaded the routing tables
> Sep  3 07:12:07 manch2 kernel: carp0: link state changed to DOWN
> Sep  3 07:12:07 manch2 kernel: carp0: link state changed to DOWN
> Sep  3 07:12:07 manch2 kernel: nfe0: link state changed to DOWN
> Sep  3 07:12:07 manch2 kernel: carp0: link state changed to DOWN
> Sep  3 07:12:11 manch2 kernel: nfe0: link state changed to UP   
> Sep  3 07:12:11 manch2 kernel: carp0: link state changed to DOWN
> Sep  3 07:12:14 manch2 kernel: carp0: link state changed to UP

Hmm, it does not look right, carp0 showed link DOWN message four
times in a row.
By the way, are you using IPMI on MCP55? nfe(4) is not ready to
handle MAC operation with IPMI.