Ethernet interface Watchdog timeout

[Aristedes Maniatis]

Last night I needed to reboot switches connected to a FreeBSD server.
There are two igb interfaces, bound via lagg0 as an LACP pair. Each is
connected to a different switch and those switches support mlag (LAG
distributed across more than one switch unit). One of the interfaces
came back fine when its switch rebooted, but when the second switch was
rebooted several hours later the other interface didn't. Both igb0 and
igb1 interfaces are on the motherboard itself.

This has happened once before, and rebooting the FreeBSD server resolved
it. Obviously I'd like to understand the problem better first. Is there
more debugging I could collect while the server is in this state?

Physically removing the ethernet cable and plugging it back in does not
bring the interface up. ifconfig down and up also does not help.

What is this watchdog timeout that we are seeing in the logs?


Ari



# ifconfig igb0
igb0: flags=8c03<UP,BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
   
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:00:ea:b2
    media: Ethernet autoselect
    status: no carrier
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


# uname -a
FreeBSD lash.internal 12.1-RELEASE-p2 FreeBSD 12.1-RELEASE-p2 GENERIC  amd64


# grep igb0 /var/log/messages
Jul  8 23:00:43 lash kernel: igb0: Watchdog timeout (TX: 0 desc avail:
42 pidx: 1003) -- resetting
Jul  8 23:00:43 lash kernel: igb0: link state changed to DOWN
Jul  8 23:00:44 lash kernel: igb0: Watchdog timeout (TX: 7 desc avail:
1024 pidx: 0) -- resetting
Jul  9 00:00:01 lash kernel: igb0: Watchdog timeout (TX: 7 desc avail:
1024 pidx: 0) -- resetting
Jul  9 05:01:12 lash kernel: igb0: Watchdog timeout (TX: 7 desc avail:
1024 pidx: 0) -- resetting
Jul  9 05:06:56 lash kernel: igb0: Watchdog timeout (TX: 7 desc avail:
1024 pidx: 0) -- resetting
Jul  9 14:25:33 lash kernel: igb0: Watchdog timeout (TX: 7 desc avail:
1024 pidx: 0) -- resetting
Jul  9 14:44:30 lash kernel: igb0: Watchdog timeout (TX: 7 desc avail:
1024 pidx: 0) -- resetting


igb0 at pci0:1:0:0:    class=0x020000 card=0x152115d9 chip=0x15218086
rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR NS
                 link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 ac1f6bffff00eab2
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
                     0 VFs configured out of 8 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0004
                     VF Device ID 0x1520
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 0018[1c0] = LTR 1
    ecap 000d[1d0] = ACS 1


# dmidecode -t baseboard
# dmidecode 3.2
Scanning /dev/mem for entry point.
SMBIOS 3.0 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
    Manufacturer: Supermicro
    Product Name: X10DRW-i
    Version: 1.02
    Serial Number: NM173S002991
    Asset Tag: Default string
    Features:
        Board is a hosting board
        Board is replaceable
    Location In Chassis: Default string
    Chassis Handle: 0x0003
    Type: Motherboard
    Contained Object Handles: 0

Handle 0x0021, DMI type 41, 11 bytes
Onboard Device
    Reference Designation: ASPEED Video AST2400
    Type: Video
    Status: Enabled
    Type Instance: 1
    Bus Address: 0000:05:00.0

Handle 0x0022, DMI type 41, 11 bytes
Onboard Device
    Reference Designation: Intel Ethernet i350 #1
    Type: Ethernet
    Status: Enabled
    Type Instance: 1
    Bus Address: 0000:01:00.0

Handle 0x0023, DMI type 41, 11 bytes
Onboard Device
    Reference Designation: Intel Ethernet i350 #2
    Type: Ethernet
    Status: Enabled
    Type Instance: 2
    Bus Address: 0000:01:00.1