kern/127870: HP DL320 lockup with kernel panic

Sun Oct 5 06:40:02 UTC 2008

>Number:         127870
>Category:       kern
>Synopsis:       HP DL320 lockup with kernel panic
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Oct 05 06:40:02 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Alexei Volkov
>Release:        7-STABLE
>Organization:
SoftLynx
>Environment:
FreeBSD host.softlynx.ru 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #2: Sun Sep 14 12:50:44 MSD 2008     root at host.softlynx.ru:/usr/obj/usr/src/sys/SOFTLYNX  amd64
>Description:
The server HP Proliant DL320 G5p fall into lockup periodically (uptime is varying from couple days up to a week) .

The last time I've saw the console message 

lockmgr: thread 0xffffff000427ca50 unlocking unheld lock
panic: vn_finished_write: neg cnt
cpuid=0
Sleeping thread (tid 100045, pid 46) owns a non-sleepable lock

and the server does not respond to any external events (keyboard, network pings and even iLO web or ssh interface)

I am not shure but it can be related with PR 123768
>How-To-Repeat:
Install FreeBSD 7-STABLE on HP DL320. Add hw.bge.allow_asf="1" into /boot/loader.conf to let the iLO work together with OS and wait until the lockup will take place.

I am not sure how it correlate with CPU and IO activity but usually it happens at night when the activity is much lower than in work hours.
>Fix:

Patch attached with submission follows:

Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-PRERELEASE #2: Sun Sep 14 12:50:44 MSD 2008
    root at host.softlynx.ru:/usr/obj/usr/src/sys/SOFTLYNX
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU            3065  @ 2.33GHz (2327.51-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6fb  Stepping = 11
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe3fd<SSE3,RSVD2,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 2
usable memory = 4279271424 (4081 MB)
avail memory  = 4116807680 (3926 MB)
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <HP ProLiant> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci21: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci18: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib4
bge0: <HP NC326i Dual Port PCI-E Gigabit Adapter, ASIC rev. 0x9003> at device 4.0 on pci3
miibus0: <MII bus> on bge0
brgphy0: <BCM5714 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:1f:29:c3:7c:ec
bge0: [ITHREAD]
bge1: <HP NC326i Dual Port PCI-E Gigabit Adapter, ASIC rev. 0x9003> mem 0xfdff0000-0xfdffffff,0xfdfe0000-0xfdfeffff irq 17 at device 4.1 on pci3
miibus1: <MII bus> on bge1
brgphy1: <BCM5714 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:1f:29:c3:7c:ed
bge1: [ITHREAD]
pcib5: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci4: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 28.3 on pci0
pci7: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 28.4 on pci0
pci10: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 28.5 on pci0
pci13: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib9
vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem 0xd8000000-0xdfffffff,0xfdef0000-0xfdefffff irq 21 at device 2.0 on pci1
pci1: <base peripheral> at device 4.0 (no driver attached)
pci1: <base peripheral> at device 4.2 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0x3800-0x381f irq 23 at device 4.4 on pci1
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <(0x103c) UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
ipmi0: <IPMI System Interface> mem 0xfddf0000-0xfddf00ff irq 22 at device 4.6 on pci1
ipmi0: using KSC interface
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH9 SATA300 controller> port 0x1080-0x1087,0x1088-0x108b,0x1090-0x1097,0x1098-0x109b,0x10a0-0x10af,0x10b0-0x10bf irq 20 at device 31.2 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
atapci1: <Intel ICH9 SATA300 controller> port 0x10c0-0x10c7,0x10c8-0x10cb,0x10d0-0x10d7,0x10d8-0x10db,0x10e0-0x10ef,0x10f0-0x10ff irq 20 at device 31.5 on pci0
atapci1: [ITHREAD]
ata4: <ATA channel 0> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci1
ata5: [ITHREAD]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
cpu0: <ACPI CPU> on acpi0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 72a072a0600072a
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 72a072a0600072a
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse Explorer, device ID 4
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
sio0: [FILTER]
ipmi1: <IPMI System Interface> on isa0
device_attach: ipmi1 attach returned 16
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1 at port 0x2f8-0x2ff irq 3 flags 0x10 on isa0
sio1: type 16550A
sio1: [FILTER]
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ukbd0: <HP Virtual Keyboard, class 0/0, rev 1.10/0.02, addr 2> on uhub0
kbd2 at ukbd0
ums0: <HP Virtual Keyboard, class 0/0, rev 1.10/0.02, addr 2> on uhub0
ums0: 3 buttons.
uhub1: <HP Virtual Hub, class 9/0, rev 1.10/0.01, addr 3> on uhub0
uhub1: 7 ports with 7 removable, self powered
Timecounters tick every 1.000 msec
ad4: 238475MB <GB0250C8045 HPG1> at ata2-master SATA150
ad6: 238475MB <GB0250C8045 HPG1> at ata3-master SATA150
GEOM_MIRROR: Device mirror/boot0 launched (2/2).
GEOM_MIRROR: Device mirror/vol0 launched (2/2).
GEOM_JOURNAL: Journal 3098540187: mirror/vol0a contains data.
GEOM_JOURNAL: Journal 3098540187: mirror/vol0a contains journal.
GEOM_JOURNAL: Journal mirror/vol0a clean.
acd0: DVDR <TEAC DV-W28E-RW/G.B1> at ata4-master UDMA33
ipmi0: IPMI device rev. 1, firmware rev. 1.2, version 2.0
ipmi0: Number of channels 8
ipmi0: Attached watchdog
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/mirror/vol0a.journal

>Release-Note:
>Audit-Trail:
>Unformatted: