Kernel Panic (Fatal trap 12: page fault while in kernel mode)

Dan D Niles dan at more.net
Thu Jun 14 20:58:18 UTC 2007


I'm not sure if this is a new issue or an existing one.  I see some
similar issues, but I can't tell if they are the same.  If this is a new
issue, I can submit a PR.

I have 4 identical Dell 2650 servers.  I upgraded one of them from 6.1
to 6.2 and it started crashing.  I installed it from scratch thinking
something got hosed in the upgrade, but it still crashed.  I swapped
disks with one of the other 6.1 machines to rule out hardware.  FreeBSD
6.2 is still crashing on different hardware (except the OS disk).

I enabled crash dumps.  Here is some output from kgdb:

# kgdb kernel.debug /var/crash/vmcore.0 
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address   = 0x104
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc066c761
stack pointer           = 0x28:0xe4f8ec90
frame pointer           = 0x28:0xe4f8ec9c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = resume, IOPL = 0
current process         = 5 (thread taskq)
trap number             = 12
panic: page fault
cpuid = 1
Uptime: 1d6h4m40s
Dumping 2047 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 2047MB (524016 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15

#0  doadump () at pcpu.h:165
165             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc067553a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0675861 in panic (fmt=0xc08e4721 "%s")
    at /usr/src/sys/kern/kern_shutdown.c:565
#3  0xc088e2dc in trap_fatal (frame=0xe4f8ec50, eva=260)
    at /usr/src/sys/i386/i386/trap.c:837
#4  0xc088da4e in trap (frame=
      {tf_fs = -969015288, tf_es = -942342104, tf_ds = -453509080, tf_edi = -969003008, tf_esi = 4, tf_ebp = -453448548, tf_isp = -453448580, tf_ebx = -955327240, tf_edx = 6, tf_ecx = 0, tf_eax = 1, tf_trapno = 12, tf_err = 0, tf_eip = -1067006111, tf_cs = 32, tf_eflags = 65538, tf_esp = -955736776, tf_ss = 4})
    at /usr/src/sys/i386/i386/trap.c:270
#5  0xc0879d8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6  0xc066c761 in _mtx_lock_sleep (m=0xc70edcf8, tid=3325964288, opts=0, 
    file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:546
#7  0xc06bbac6 in unp_gc (arg=0x0, pending=2)
    at /usr/src/sys/kern/uipc_usrreq.c:1714
#8  0xc06961e3 in taskqueue_run (queue=0xc64b0200)
    at /usr/src/sys/kern/subr_taskqueue.c:257
#9  0xc06966c6 in taskqueue_thread_loop (arg=0x1)
    at /usr/src/sys/kern/subr_taskqueue.c:376
#10 0xc065ec7d in fork_exit (callout=0xc0696634 <taskqueue_thread_loop>, 
    arg=0xc09e3ca8, frame=0xe4f8ed38) at /usr/src/sys/kern/kern_fork.c:821
---Type <return> to continue, or q <return> to quit---
#11 0xc0879dec in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208

Is the Undefined symbol "ps_pglobal_lookup" at the start of the kgdb output a problem?

I'd like to upgrade all 4 servers to 6.2, but I need to resolve this
first.

What can I do to help debug this?  I have the vmcore.0 file, and I can
crash the box within a day or two.

Thanks,

Dan

PS Please be sure to cc me, I'm not on the list.




More information about the freebsd-bugs mailing list