Some Broadcom GbE NIC(bge driver) suffers packet loss at receiving

Thu Aug 16 02:56:56 PDT 2007

Hi.

Some revision of Broadcom GbE NIC with bge driver 
seems to loss 1 packet per 1000 packet at receiving on some occasion. 
Sending doesn't have this problem(receiving only).

This loss doesn't always happen, 
but when happens, it seems that loss is exactly 1 per 1000, 
as the last 3 digits of lost packets' sequence number are all same.
(i.e. example of lost packet seq-No. are:  
 3284, 4284, 5284, 8284, 9284, ...)

I've tested with:
  * tcpdump capturing of SmartBits, 
  * ports/net/rude(network traffic generator/collector software) 
  * owping(one-way-ping)

This loss happens with:
  * BCM5704(?) @ FreeBSD/i386 6-stable (Aug. 2007)
    (6-stable of some months ago also suffered with this problem)
  * same type hardware
    (I tested with 2 HP DL385 servers, and happend with both H/W.)

This loss doesn't happen with:
  * same hardware with Linux(Fedora-Core 5)
  * same server with different NIC (Intel GbE with em driver) 
    also with same version of FreeBSD/i386 6-stable
  * same FreeBSD/i386 6-stable with some other Broadcom cards(Broadcom BCM5752)
    (not same PC/server, but other PC hardware)

-------------------------------------------------------
More precise hardware information is given below:

This happens with BCM5704 of this revision:
  (on-board with HP DL385 Opteron Server)

bge0: <Broadcom BCM5704 B0, ASIC rev. 0x2100> mem 0xf7df0000-0xf7dfffff irq 28 at device 6.0 on pci3
bge0: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xf7df0000
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus0
brgphy0: OUI 0x000818, model 0x0019, rev. 0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: bpf attached
bge0: Ethernet address: 00:17:a4:8f:22:bc
ioapic2: routing intpin 0 (PCI IRQ 28) to vector 53
bge0: [MPSAFE]
bge1: <Broadcom BCM5704 B0, ASIC rev. 0x2100> mem 0xf7de0000-0xf7deffff irq 29 at device 6.1 on pci3
bge1: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xf7de0000
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus1
brgphy1: OUI 0x000818, model 0x0019, rev. 0
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: bpf attached
bge1: Ethernet address: 00:17:a4:8f:22:bb
ioapic2: routing intpin 1 (PCI IRQ 29) to vector 54
bge1: [MPSAFE]

pciconf -lvc:
bge0 at pci3:6:0:  class=0x020000 card=0x00d00e11 chip=0x164814e4 rev=0x10 hdr=0x00
    vendor     = 'Broadcom Corporation'
    device     = 'BCM5704 NetXtreme Dual Gigabit Adapter'
    class      = network
    subclass   = ethernet
    cap 07[40] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction
    cap 01[48] = powerspec 2  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 05[58] = MSI supports 8 messages, 64 bit 

Other bge cards that doesn't suffer with this problem:
  (HP xw8400 Workstation on-board)

bge0: <Broadcom BCM5752 A1, ASIC rev. 0x6001> mem 0xfb600000-0xfb60ffff irq 17 at device 0.0 on pci31
bge0: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xfb600000
bge0: Disabling fastboot
bge0: Disabling fastboot
miibus0: <MII bus> on bge0
brgphy0: <BCM5752 10/100/1000baseTX PHY> on miibus0
brgphy0: OUI 0x000818, model 0x0010, rev. 0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: bpf attached
bge0: Ethernet address: 00:17:08:2a:a3:a3
ioapic0: routing intpin 17 (PCI IRQ 17) to vector 49
bge0: [MPSAFE]

pciconf -lvc:
bge0 at pci31:0:0: class=0x020000 card=0x3015103c chip=0x160014e4 rev=0x01 hdr=0x00
    vendor     = 'Broadcom Corporation'
    device     = 'Broadcom NetXtreme Gigabit Ethernet'
    class      = network
    subclass   = ethernet
    cap 01[48] = powerspec 2  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 05[58] = MSI supports 8 messages, 64 bit 
    cap 10[d0] = PCI-Express 1 endpoint

-------------------------------------------------------

#######################################################

Example usage of ports/net/rude (and crude) to test this.

#######################################################
(At packet sender:)

sender% cat > rude.cfg << EOFEOF
START NOW
1000 0030 ON 3002 10.1.1.1:10001 CONSTANT 10000 200
11000 0030 OFF
EOFEOF
sender% 
	# Above config will send 10000 packets/sec for 10 secs.
	# ==> 100001 packets will be generated.
	#     (+1 comes from rude's spec(bug?))
sender% rude -s rude.cfg				(Run packet generator)
rude version 0.70, Copyright (C) 1999 Juha Laine and Sampo Saaristo
rude comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under GNU GENERAL PUBLIC LICENSE Version 2.

F_ID: F_START: F_STOP: F_SPORT: F_DADD: F_DPORT: F_err: F_suc: F_seq: F_TYPE: [+ type params]
30 1187254729.594905 1187254739.594905 3002 129.60.75.70 10001 0 100001 100001 CBR [r:10000 s:200]
sender% 

#######################################################
(At packet receiver; which have bge cards)

receiver% crude -s 30
crude version 0.70, Copyright (C) 1999 Juha Laine and Sampo Saaristo
crude comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under GNU GENERAL PUBLIC LICENSE Version 2.
^C						(Press Ctrl+C after rude at sender has finished)
Runtime statistics results: 
--------------------------- 

Flow_ID=30 
Packets: received=100001   out-of-seq=0   lost(est)=0 
Total bytes received=20000200 
Sequence numbers: first=0   last=100000 
Delay: average = -0.001333   jitter=0.000069   seconds 
Absolute maximum jitter=0.008760   seconds 
Throughput=2.00001e+06   Bps  (from first to last packet received) 
receiver% 

-------------------------------------------------------

For good GbE-cards, packet received count should be equal with 
sender's packet count.
For some bge cards mentioned above, lost packet percentage 
comes ~ 0.1%, and when looking to sequence numbers,
the loss seem to happen every 1000 packets.