panic on 6.0 with amd64

[Howard]

Hi all,

I'm experiencing kernel panics on a webserver running freebsd 6.0.  The
panics are infrequent (about once a month) and do occur under mild load
(even under only 0.4 load).  Following the handbook's instructions I got the
dump info and also made the dmesg available.  I apologize in advance, I'm
extremely new to this type of problem so if any information is missing
please let me know.  Thanks for any help or tips, they are much appreciated.

Here is the dump info:


# kgdb kernel.debug.0 vmcore.3
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:
            Sep  5 16:14:02 aeon sm-mta[37276]: k85L6atQ032754:
SYSERR(root): deliver: mci=642018 rcode=0 errno=0 state=0 sig=
relay.verizon.net.                            Sep  5 16:44:03 aeon
sm-mta[38256]: k85L1oiE032593: SYSERR(root): deliver: mci=643018 rcode=0
errno=0 state=0 sig=relay.verizon.net.                            Sep  5
17:14:20 aeon sm-mta[39178]: k85JucY6030388: SYSERR(root): deliver:
mci=643018 rcode=0 errno=0 state=0 sig=relay.verizon.net.
Sep  5 17:45:31 aeon sm-mta[40231]: k85IWEZL027822: S

#0  doadump () at pcpu.h:172
172     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) backtrace
#0  doadump () at pcpu.h:172
#1  0x0000000000000004 in ?? ()
#2  0xffffffff803b50c3 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:399
#3  0xffffffff803b56c6 in panic (fmt=0xffffff007b9ed720 "@ÓÊa") at
/usr/src/sys/kern/kern_shutdown.c:555
#4  0xffffffff8058bfbf in trap_fatal (frame=0xffffff007b9ed720,
eva=18446742975838606144) at /usr/src/sys/amd64/amd64/trap.c:655
#5  0xffffffff8058c462 in trap (frame=
      {tf_rdi = -140737488355328, tf_rsi = -1099450764336, tf_rdx =
6053253321325322206, tf_rcx = 34369499136, tf_r8 = 0, tf_r9 =
-1097427386368, tf_rax = 0, tf_rbx = -1384231984, tf_rbp =
6053253321325322206, tf_r10 = 6385350, tf_r11 = 425690, tf_r12 =
-1099450764416, tf_r13 = -1097376303568, tf_r14 = 140737488355328, tf_r15 =
0, tf_trapno = 27, tf_addr = 0, tf_flags = -1396024512, tf_err = 0, tf_rip =
-2141685473, tf_cs = 8, tf_rflags = 66054, tf_rsp = -1310041440, tf_ss =
16}) at /usr/src/sys/amd64/amd64/trap.c:467
#6  0xffffffff8057bb6b in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:168
#7  0xffffffff8058791f in pmap_remove_pages (pmap=0xffffff0003a0b380, sva=0,
eva=140737488355328) at /usr/src/sys/amd64/amd64/pmap.c:2531
#8  0xffffffff80399561 in exit1 (td=0xffffff007b9ed720, rv=0) at
vm_map.h:252
#9  0xffffffff8039987e in sys_exit (td=0xffff800000000000,
uap=0xffffff0003a0b3d0) at /usr/src/sys/kern/kern_exit.c:97
#10 0xffffffff8058cdd1 in syscall (frame=
      {tf_rdi = 0, tf_rsi = 34365342200, tf_rdx = 34369564640, tf_rcx = 0,
tf_r8 = 0, tf_r9 = 0, tf_rax = 1, tf_rbx = 0, tf_rbp = 0, tf_r10 = 0, tf_r11
= 0, tf_r12 = 5628032, tf_r13 = 5533792, tf_r14 = 43, tf_r15 =
140737488349504, tf_trapno = 12, tf_addr = 34368757184, tf_flags = 12,
tf_err = 2, tf_rip = 34369078716, tf_cs = 43, tf_rflags = 518, tf_rsp =
140737488348776, tf_ss = 35}) at /usr/src/sys/amd64/amd64/trap.c:787
#11 0xffffffff8057bd08 in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:270
#12 0x00000008008e85bc in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit



and dmesg:



Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 6.0-RELEASE #0: Wed Nov  2 19:07:38 UTC 2005
    root at rat.samsco.home:/usr/obj/usr/src/sys/GENERIC
ACPI APIC Table: <A M I  OEMAPIC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Opteron(tm) Processor 244 (1792.82-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0xf51  Stepping = 1

Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow+,3DNow>
real memory  = 2147418112 (2047 MB)
avail memory = 2063310848 (1967 MB)
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-27 on motherboard
ioapic2 <Version 1.1> irqs 28-31 on motherboard
acpi0: <A M I OEMRSDT> on motherboard
acpi0: Power Button (fixed)
pci_link0: <ACPI PCI Link LNKA> irq 5 on acpi0
pci_link1: <ACPI PCI Link LNKB> irq 9 on acpi0
pci_link2: <ACPI PCI Link LNKC> irq 11 on acpi0
pci_link3: <ACPI PCI Link LNKD> irq 10 on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x5008-0x500b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci3: <ACPI PCI bus> on pcib1
ohci0: <OHCI (generic) USB controller> mem 0xfeafc000-0xfeafcfff irq 19 at
device 0.0 on pci3
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xfeafd000-0xfeafdfff irq 19 at
device 0.1 on pci3
ohci1: [GIANT-LOCKED]
usb1: OHCI version 1.0, legacy support
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
atapci0: <SiI 3114 SATA150 controller> port
0xbc00-0xbc07,0xb400-0xb403,0xb000-0xb007,0xac00-0xac03,0xa800-0xa80f mem
0xfeafec00-0xfeafefff irq 19 at device 5.0 on pci3
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
ata4: <ATA channel 2> on atapci0
ata5: <ATA channel 3> on atapci0
pci3: <display, VGA> at device 6.0 (no driver attached)
fxp0: <Intel 82551 Pro/100 Ethernet> port 0xa400-0xa43f mem
0xfeafb000-0xfeafbfff,0xfeaa0000-0xfeabffff irq 18 at device 8.0 on pci3
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:e0:81:2a:15:a9
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <AMD 8111 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
ata0: <ATA channel 0> on atapci1
ata1: <ATA channel 1> on atapci1
pci0: <serial bus, SMBus> at device 7.2 (no driver attached)
pci0: <bridge> at device 7.3 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci2: <ACPI PCI bus> on pcib2
bge0: <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2003> mem
0xfc8c0000-0xfc8cffff,0xfc8b0000-0xfc8bffff irq 24 at device 9.0 on pci2
miibus1: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus1
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
bge0: Ethernet address: 00:e0:81:2a:50:3e
bge1: <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2003> mem
0xfc8f0000-0xfc8fffff,0xfc8e0000-0xfc8effff irq 25 at device 9.1 on pci2
miibus2: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus2
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
bge1: Ethernet address: 00:e0:81:2a:50:3f
pci0: <base peripheral, interrupt controller> at device 10.1 (no driver
attached)
pcib3: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci1: <ACPI PCI bus> on pcib3
pci0: <base peripheral, interrupt controller> at device 11.1 (no driver
attached)
acpi_button0: <Power Button> on acpi0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcc7ff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1792816307 Hz quality 800
Timecounters tick every 1.000 msec
acd0: CDROM <QSI CD-ROM SCR-242/CXAC> at ata0-master UDMA33
ad10: 238475MB <Seagate ST3250823AS 3.03> at ata5-master SATA150