Problems with bge (possibly related to r208993)

Tue Jun 15 03:09:11 UTC 2010

On Tuesday 15 June 2010 01:03:43 Pyun YongHyeon wrote:
> On Sun, Jun 13, 2010 at 07:34:11PM +0400, Artem Kim wrote:
> > Hi,
> >
> > I have two routers (HP DL140G3):
> >
> > NAS3 FreeBSD 8.1-PRERELEASE # 0: Thu Jun 3 04:13:07 MSD 2010 i386
> > NAS2 FreeBSD 8.1-PRERELEASE # 0: Sat Jun 12 16:42:19 UTC 2010 i386
> > (r208993 included)
> >
> > bge0 @ pci0: 19:0:0: class = 0x020000 card = 0x3260103c chip = 0x165914e4
> > rev = 0x11 hdr = 0x00
> >     vendor = 'Broadcom Corporation'
> >     device = 'NetXtreme Gigabit Ethernet PCI Express (BCM5721)'
> >     class = network
> >     subclass = ethernet
> > bge1 @ pci0: 20:0:0: class = 0x020000 card = 0x3260103c chip = 0x165914e4
> > rev = 0x11 hdr = 0x00
> >     vendor = 'Broadcom Corporation'
> >     device = 'NetXtreme Gigabit Ethernet PCI Express (BCM5721)'
> >     class = network
> >     subclass = ethernet
> >
> >
> > I have some problems with bge on NAS2.
> >
> > After some time (about 15 hours) bge1 stops flowing traffic.
> > NAS3 NAS3 - pppoe server. Through bge1 passes only ip traffic through
> > bge0 no ip-traffic.
> > Problems occur only with the bge1 interface on NAS2.
> >
> >
> > Traffic through bge1 not pass until I will not do "ifconfig bge1 down
> > ifconfig bge1 up".
> >
> > When I do "ifconfig bge0 down" NIC does not shutdown:
> >
> > nas2 # ifconfig bge1 down
> > nas2 #
> > nas2 # ifconfig bge1
> > bge1: flags = 8843 <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> > 1500 options = 8009b
> > <RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
> >         ether XXXXXXXXXXXXX
> >         inet YYYYYYYYYYY netmask 0xffffffc0 broadcast YYYYYYYYYYYY
> >         media: Ethernet autoselect (1000baseT <full-duplex>)
> >         status: active
> >
> > LED also indicates that the NIC is active.
> >
> > I left the NAS in a state of "frozen bge1" - and can provide additional
> > information for diagnosis.
> 
> Try run tcpdump on bge1 and see whether driver still see incoming
> traffic. Also show me the output of "netstat -ndI bge1" and output
> of "sysctl dev.bge.1.stats". Verbose dmesg output also would be
> helpful.

nas2 # netstat-ndI bge1
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
bge1 1500 <Link#3> 00:1 b: 78: a3: 3c: 01 418543876 1972918 0 446063237 0 0 0
bge1 1500 XX.XX.6.12 XX.XX.6.133 890,306 - - 1,076,833 - - -

Should I add additional debugging options?

nas2 # sysctl dev.bge.1.stats
sysctl: unknown oid 'dev.bge.1.stats'

nas2 # sysctl dev.bge.1
dev.bge.1.% desc: Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 
0x004101
dev.bge.1.% driver: bge
dev.bge.1.% location: slot = 0 function = 0
dev.bge.1.% pnpinfo: vendor = 0x14e4 device = 0x1659 subvendor = 0x103c 
subdevice = 0x3260 class = 0x020000
dev.bge.1.% parent: pci20
dev.bge.1.forced_collapse: 0

I can show verbose dmesg, but this requires a reboot so bge1 come out of the 
current state.

I looked tcpdump on NAS2 - and I only saw the ARP requests from NAS2 (NAS2 - 
XX.XX.6.133):

nas2 # tcpdump -i bge1
tcpdump: verbose output suppressed, use-v or-vv for full protocol decode
listening on bge1, link-type EN10MB (Ethernet), capture size 96 bytes
01:23:43.063238 ARP, Request who-has XX.XX.6.129 tell XX.XX.6.133, length 28
01:23:43.162257 ARP, Request who-has XX.XX.6.129 tell XX.XX.6.133, length 28
01:23:43.935016 ARP, Request who-has XX.XX.6.129 tell XX.XX.6.133, length 28

XX.XX.6.129 is l3-switch(AT-X900) default router for NAS2; bge1 is directly 
connected to the x900.

I looked tcpdump on the x900:

awplus # tcpdump -ni vlanXX host XX.XX.6.133
05:36:30.455642 arp who-has XX.XX.6.129 tell XX.XX.6.133
05:36:30.455898 arp reply XX.XX.6.129 is-at 00:00: cd: 29:6 e: 09
05:36:31.483353 arp who-has XX.XX.6.129 tell XX.XX.6.133
05:36:31.483505 arp reply XX.XX.6.129 is-at 00:00: cd: 29:6 e: 09
05:36:32.511260 arp who-has XX.XX.6.129 tell XX.XX.6.133
05:36:32.511353 arp reply XX.XX.6.129 is-at 00:00: cd: 29:6 e: 09
05:36:33.539163 arp who-has XX.XX.6.129 tell XX.XX.6.133

ARP requests from NAS2 (XX.XX.6.133). But on NAS2 I can _only_ see ARP-
requests from NAS2.

I added static arp-entry on NAS2 and do ping XX.XX.6.129.

Then I looked again at tcpdump on XX.XX.6.129:

awplus # tcpdump -nei vlanXX host XX.XX.6.133
06:13:03.472539 00:00: cd: 29:6 e: 09> ff: ff: ff: ff: ff: ff, ethertype ARP 
(0x0806), length 42: arp who-has XX.XX.6.133 tell XX. XX.6.129
06:13:03.526768 00:1 b: 78: a3: 3c: 01> 00:00: cd: 29:6 e: 09, ethertype IPv4 
(0x0800), length 98: XX.XX.6.133> XX.XX.6.129: ICMP echo request, id 6958, seq 
1920, length 64
06:13:04.553728 00:1 b: 78: a3: 3c: 01> 00:00: cd: 29:6 e: 09, ethertype IPv4 
(0x0800), length 98: XX.XX.6.133> XX.XX.6.129: ICMP echo request, id 6958, seq 
1921, length 64
06:13:04.554495 00:00: cd: 29:6 e: 09> ff: ff: ff: ff: ff: ff, ethertype ARP 
(0x0806), length 42: arp who-has XX.XX.6.133 tell XX. XX.6.129
06:13:05.554486 00:00: cd: 29:6 e: 09> ff: ff: ff: ff: ff: ff, ethertype ARP 
(0x0806), length 42: arp who-has XX.XX.6.133 tell XX. XX.6.129
06:13:05.581488 00:1 b: 78: a3: 3c: 01> 00:00: cd: 29:6 e: 09, ethertype IPv4 
(0x0800), length 98: XX.XX.6.133> XX.XX.6.129: ICMP echo request, id 6958, seq 
1922, length 64

Traffic comes from XX.XX.6.133 to XX.XX.6.129. I tried to add arp-entry on 
XX.XX.6.129 (XX.XX.6.133 -> 001b.78a3.3c01) but communication does not 
restore.