Rare / random hangs with FreeBSD 6.2-RELEASE-p4

Tue Sep 18 04:32:59 PDT 2007

Hi

Running a FreeBSD 6.2 with mainly MySQL, Apache and PHP (kept up to date
with manual freebsd-update / portupgrade).

The hardware is SuperMicro SuperServer 5015MT+ with 2GB of ECC RAM and a
Intel Core 2 Quad 6700. Running a BIOS assisted soft mirror with SATA
disks...

After aprox 2 weeks the server hangs. There is no dumps at /var/crash as the
server seems to lock up while dumping. Details below - unfortunately I don't
always get a picture of the console as I'm not always the rebooter.
There is no hang-related notes i FreeBSD logs, and nothing at all in IPMI
and BIOS logs.

Earlier I had the server running with HTTP Accept Filters on, but then it
reboots 4-5 times a day! I've also been running xcache (php accelrator), but
that was disabled during the latest hang.

Do You have any suggestions?
Does this seems to be hardware or software related?
Anything I could try to figure out what is going on?
Unfortunately the server is in production.

Thank You for any help.

Regards
    Gert Lynge

**** Hang 1
panic: vm_fault: fault on nofault entry, addr: e92cf000
cpuid = 3
Uptime: 10d0h24m25s
Dumping 2046 MB (2 chunks)
  chunk 0: 1MB (150 pages) ... Ok
  chunk 1: 2046MB (523744 pages)_
---
**** Hang 2:
Fatal trap 12: page fault while in kernel mode
Cupid = 2; apic id = 2
Fault virtual address = 0x5a
Fault code = supervisor read, page not present
Instruction pointer = 0x20:0xc07e40a3
Stack pointer = 0x28:0xe6a35b74
Frame pointer = 0x28:0xe6a35c40
Code segment = base 0x0, limit 0xfffff, type 0x1b
             = DPL 0, pres 1, def32 1, gran 1
Processor eflags = interrupt enabled, resume, IOPL = 0
Current process = 44 (pagezero)
Trap number = 12
Panic: page fault
Cupid = 2
Uptime 17d22h47m51s
Dumping 2048 MB (2 chunks)
  Chunk 0: 1MB (150 pages)ipfw: xxxxxxxx
ipfw: xxxxxxxxxx
---
**** uname -a
FreeBSD x.x.x 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #0: Thu Apr 26 17:55:55
UTC 2007     root at i386-builder.daemonology.net:/usr/obj/usr/src/sys/SMP
i386
---
**** cat /etc/rc.conf
defaultrouter="x.x.x.x"
font8x14="cp865-8x14"
font8x16="cp865-8x16"
font8x8="cp865-8x8"
hostname="x.x.x"
ifconfig_em1="inet x.x.x.x  netmask x.x.x.x"
saver="daemon"
usbd_enable="YES"
keymap="danish.cp865"
keyrate="fast"
sshd_enable="YES"
firewall_enable="YES"
firewall_type="x"
firewall_logging="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
mysql_enable="YES"
apache22_enable="YES"
#apache22_http_accept_enable="YES"
inetd_enable="YES"
clear_tmp_enable="YES"
#log_in_vain="1"
sendmail_enable="YES"
rsyncd_enable="YES"
syslogd_flags="-a x.x.x.x/x:*"
clamav_freshclam_enable="YES"
local_startup="/usr/local/etc/rc.d"
dumpdev="/dev/ar0s1b"
---
**** cat /var/run/dmesg.boot
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p4 #0: Thu Apr 26 17:55:55 UTC 2007
    root at i386-builder.daemonology.net:/usr/obj/usr/src/sys/SMP
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU    Q6700  @ 2.66GHz (2660.01-MHz 686-class
CPU)
  Origin = "GenuineIntel"  Id = 0x6fb  Stepping = 11

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA
,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>>
  AMD Features=0x20000000<LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4
real memory  = 2146304000 (2046 MB)
avail memory = 2095165440 (1998 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci9: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci9
pci10: <ACPI PCI bus> on pcib3
pci9: <base peripheral, interrupt controller> at device 0.1 (no driver
attached)
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci13: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x4000-0x401f mem 0xe0200000-0xe021ffff irq 16 at device 0.0 on pci13
em0: Ethernet address: 00:30:48:8d:1f:5e
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci14: <ACPI PCI bus> on pcib5
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x5000-0x501f mem 0xe0300000-0xe031ffff irq 17 at device 0.0 on pci14
em1: Ethernet address: 00:30:48:8d:1f:5f
uhci0: <UHCI (generic) USB controller> port 0x3000-0x301f irq 23 at device
29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x3020-0x303f irq 19 at device
29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x3040-0x305f irq 18 at device
29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x3060-0x307f irq 16 at device
29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xe0000000-0xe00003ff
irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci15: <ACPI PCI bus> on pcib6
pci15: <display, VGA> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
atapci1: <Intel ICH7 SATA300 controller> port
0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem
0xe0000400-0xe00007ff irq 19 at device 31.2 on pci0
atapci1: AHCI Version 01.10 controller with 4 ports detected
ata2: <ATA channel 0> on atapci1
ata3: <ATA channel 1> on atapci1
ata4: <ATA channel 2> on atapci1
ata5: <ATA channel 3> on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
ppc0: <ECP parallel printer port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3
on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
pmtimer0 on isa0
ipmi0: <IPMI System Interface> on isa0
ipmi0: KCS mode found at io 0xca8 alignment 0x4 on isa
orm0: <ISA Option ROMs> at iomem
0xc0000-0xcafff,0xcb000-0xcf7ff,0xcf800-0xd07ff,0xd0800-0xd17ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: CDROM <CD-224E-N/1.AA> at ata0-master UDMA33
ad4: 76319MB <Seagate ST380811AS 3.AAE> at ata2-master SATA300
ad6: 76319MB <Seagate ST380811AS 3.AAE> at ata3-master SATA300
ipmi0: IPMI device rev. 0, firmware rev. 2.2, version 2.0
ipmi0: Number of channels 4
ipmi0: Attached watchdog
ar0: 76316MB <Intel MatrixRAID RAID1> status: READY
ar0: disk0 READY (master) using ad4 at ata2-master
ar0: disk1 READY (mirror) using ad6 at ata3-master
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/ar0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled,
default to deny, logging disabled