kern/152294: [bge] 8.1-R driver stop to reveice incoming packets on high bandwidth for BCM5784, 7.3-R works

Mykola Zubach zuborg at gmail.com
Tue Nov 16 13:10:06 UTC 2010 

>Number:         152294
>Category:       kern
>Synopsis:       [bge] 8.1-R driver stop to reveice incoming packets on high bandwidth for BCM5784, 7.3-R works
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Nov 16 13:10:04 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Mykola Zubach
>Release:        8.1-RELEASE
>Organization:
AdvancedHosters.com
>Environment:
FreeBSD 23382 7.3-RELEASE-p2 FreeBSD 7.3-RELEASE-p2 #0: Fri Jul 16 14:22:06 UTC 2010     root at DS1105:/usr/obj/usr/src/sys/Z-AMD64  amd64
>Description:
On 8.1-RELEASE (both custom and GENERIC kernels) after some period of quite high bandwidth (10-20 minutes of 600-800Mbit/s) bge0 device fail to receive any incoming packets.

After such fail 'tcpdump' show no incoming traffic, ssh still provide some amount of outgoing data (like 'top' or 'nload' output), but keypresses don't reach the server and tcp session drops after timeout.

These two commands restore working state
# ifconfig bge0 down
# ifconfig bge0 up

Without 'polling' driver hangs up very quicky, in 1-2 minutes. Active 'polling' can provide 10-20 minutes of networking.

Bandwidth is produced by 'nginx' webserver. Various configurations were tested, like 'aio sendfile' feature, or pure read/write operations without 'sendfile on' - in all cases bge fails.

Kernel is compiled with ZERO_COPY_SOCKETS option, but GENERIC kernel fails too.

7.3-RELEASE works fine under same high load.

# dmesg
FreeBSD 8.1-RELEASE-p1 #0: Mon Oct 18 11:31:13 UTC 2010
    root at DS1728:/usr/obj/usr/src/sys/Z-AMD64 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz (2394.00-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x106e5  Family = 6  Model = 1e  Stepping = 5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x98e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 17179869184 (16384 MB)
avail memory = 16546603008 (15780 MB)
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  2
 cpu2 (AP): APIC ID:  4
 cpu3 (AP): APIC ID:  6
ioapic0 <Version 2.0> irqs 0-23 on motherboard
netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling
kbd1 at kbdmux0
acpi0: <HP ProLiant> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci0: <base peripheral> at device 8.0 (no driver attached)
pci0: <base peripheral> at device 8.1 (no driver attached)
pci0: <base peripheral> at device 8.2 (no driver attached)
pci0: <base peripheral> at device 8.3 (no driver attached)
pci0: <base peripheral> at device 16.0 (no driver attached)
pci0: <base peripheral> at device 16.1 (no driver attached)
ehci0: <Intel PCH USB 2.0 controller USB-B> mem 0xdfd02000-0xdfd023ff irq 16 at device 26.0 on pci0
ehci0: [ITHREAD]
usbus0: EHCI version 1.0
usbus0: <Intel PCH USB 2.0 controller USB-B> on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci16: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci32: <ACPI PCI bus> on pcib3
bge0: <HP NC107i PCIe Gigabit Server Adapter, ASIC rev. 0x5784100> mem 0xdf900000-0xdf90ffff irq 16 at device 0.0 on pci32
miibus0: <MII bus> on bge0
brgphy0: <BCM5784 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: d4:85:64:38:ff:70
bge0: [FILTER]
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci34: <ACPI PCI bus> on pcib4
bge1: <HP NC107i PCIe Gigabit Server Adapter, ASIC rev. 0x5784100> mem 0xdfa00000-0xdfa0ffff irq 17 at device 0.0 on pci34
miibus1: <MII bus> on bge1
brgphy1: <BCM5784 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: d4:85:64:38:ff:71
bge1: [FILTER]
pcib5: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci36: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> mem 0xde000000-0xdeffffff,0xdf800000-0xdf803fff,0xdf000000-0xdf7fffff irq 18 at device 0.0 on pci36
pcib6: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci38: <ACPI PCI bus> on pcib6
ehci1: <Intel PCH USB 2.0 controller USB-A> mem 0xdfd02400-0xdfd027ff irq 23 at device 29.0 on pci0
ehci1: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <Intel PCH USB 2.0 controller USB-A> on ehci1
pcib7: <PCI-PCI bridge> at device 30.0 on pci0
pci48: <PCI bus> on pcib7
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PCH SATA300 controller> port 0x1830-0x1837,0x1824-0x1827,0x1828-0x182f,0x1820-0x1823,0x1800-0x181f mem 0xdfd01000-0xdfd017ff irq 18 at device 31.2 on pci0
atapci0: [ITHREAD]
atapci0: AHCI called from vendor specific driver
atapci0: AHCI v1.30 controller with 6 3Gbps ports, PM supported
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci0
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci0
ata5: [ITHREAD]
ata6: <ATA channel 4> on atapci0
ata6: [ITHREAD]
ata7: <ATA channel 5> on atapci0
ata7: [ITHREAD]
ichsmb0: <Intel PCH SMBus controller> port 0x1840-0x185f mem 0xdfd02800-0xdfd028ff irq 18 at device 31.3 on pci0
ichsmb0: [ITHREAD]
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdf000-0xdffff on isa0
atkbd: unable to set the command byte.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: unable to set the command byte.
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
Timecounter "TSC" frequency 2393999100 Hz quality 800
Timecounters tick every 0.976 msec
ipfw2 initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to accept, logging disabled
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ad4: 953869MB <WDC WD1002FBYS-18W8B1 03.00C10> at ata2-master UDMA100 SATA 3Gb/s
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ad6: 953869MB <WDC WD1002FBYS-18W8B1 03.00C10> at ata3-master UDMA100 SATA 3Gb/s
GEOM: ad4s1: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad4s1: media size does not match label.
ad8: 953869MB <WDC WD1002FBYS-18W8B1 03.00C10> at ata4-master UDMA100 SATA 3Gb/s
GEOM: ad4s2: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad4s2: media size does not match label.
ad10: 953869MB <WDC WD1002FBYS-18W8B1 03.00C10> at ata5-master UDMA100 SATA 3Gb/s
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
GEOM: ad6s1: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad6s1: media size does not match label.
GEOM: ad6s2: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad6s2: media size does not match label.
GEOM: ad8s1: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad8s1: media size does not match label.
GEOM_MIRROR: Device mirror/gm0 launched (3/3).
GEOM: ad8s2: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad8s2: media size does not match label.
GEOM: ad10s1: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad10s1: media size does not match label.
GEOM_MIRROR: Device mirror/gm2 launched (1/1).
GEOM: ad10s2: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad10s2: media size does not match label.
GEOM_MIRROR: Device mirror/gm1 launched (4/4).
Root mount waiting for: usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0020, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0020, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen1.3: <ServerEngines> at usbus1
ukbd0: <ServerEngines SE USB Device, class 0/0, rev 1.10/0.01, addr 3> on usbus1
kbd2 at ukbd0
Trying to mount root from ufs:/dev/mirror/gm2a

# pciconf -lv
hostb0 at pci0:0:0:0:      class=0x060000 card=0x3318103c chip=0xd1308086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = HOST-PCI
pcib1 at pci0:0:3:0:       class=0x060400 card=0x3318103c chip=0xd1388086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-PCI
none0 at pci0:0:8:0:       class=0x088000 card=0x00000000 chip=0xd1558086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none1 at pci0:0:8:1:       class=0x088000 card=0x00000000 chip=0xd1568086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none2 at pci0:0:8:2:       class=0x088000 card=0x00000000 chip=0xd1578086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none3 at pci0:0:8:3:       class=0x088000 card=0x00000000 chip=0xd1588086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none4 at pci0:0:16:0:      class=0x088000 card=0x00000000 chip=0xd1508086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
none5 at pci0:0:16:1:      class=0x088000 card=0x00000000 chip=0xd1518086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = base peripheral
ehci0 at pci0:0:26:0:      class=0x0c0320 card=0x3118103c chip=0x3b3c8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = serial bus
    subclass   = USB
pcib2 at pci0:0:28:0:      class=0x060400 card=0x3318103c chip=0x3b428086 rev=0x05 hdr=0x01
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-PCI
pcib3 at pci0:0:28:4:      class=0x060400 card=0x3318103c chip=0x3b4a8086 rev=0x05 hdr=0x01
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-PCI
pcib4 at pci0:0:28:5:      class=0x060400 card=0x3318103c chip=0x3b4c8086 rev=0x05 hdr=0x01
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-PCI
pcib5 at pci0:0:28:6:      class=0x060400 card=0x3318103c chip=0x3b4e8086 rev=0x05 hdr=0x01
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-PCI
pcib6 at pci0:0:28:7:      class=0x060400 card=0x3318103c chip=0x3b508086 rev=0x05 hdr=0x01
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-PCI
ehci1 at pci0:0:29:0:      class=0x0c0320 card=0x3118103c chip=0x3b348086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = serial bus
    subclass   = USB
pcib7 at pci0:0:30:0:      class=0x060401 card=0x3318103c chip=0x244e8086 rev=0xa5 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801 Family (ICH2/3/4/5/6/7/8/9,63xxESB) Hub Interface to PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
isab0 at pci0:0:31:0:      class=0x060100 card=0x3118103c chip=0x3b148086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = PCI-ISA
atapci0 at pci0:0:31:2:    class=0x010601 card=0x3118103c chip=0x3b228086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'IBEX AHCI Controller(6Port)'
    class      = mass storage
    subclass   = SATA
ichsmb0 at pci0:0:31:3:    class=0x0c0500 card=0x3318103c chip=0x3b308086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = serial bus
    subclass   = SMBus
bge0 at pci0:32:0:0:       class=0x020000 card=0x705d103c chip=0x165b14e4 rev=0x10 hdr=0x00
    vendor     = 'Broadcom Corporation'
    class      = network
    subclass   = ethernet
bge1 at pci0:34:0:0:       class=0x020000 card=0x705d103c chip=0x165b14e4 rev=0x10 hdr=0x00
    vendor     = 'Broadcom Corporation'
    class      = network
    subclass   = ethernet
vgapci0 at pci0:36:0:0:    class=0x030000 card=0x31fa103c chip=0x0522102b rev=0x02 hdr=0x00
    vendor     = 'Matrox Electronic Systems Ltd.'
    device     = 'Matrox G200e (ServerEngines) - English (G200e)'
    class      = display
    subclass   = VGA

Server model: ProLiant DL120 G6
>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:

More information about the freebsd-bugs mailing list