kernel panics on a 5.4-STABLE

[Maechler Philippe]

Hello Group

We have a FreeBSD 5.4-STABLE System which is booting once or
twice in a week without any meaningfull messages on the console
or in a logfile.

May 14 11:41:26 ldxp syslogd: kernel boot file is
/boot/kernel/kernel
May 14 11:41:26 ldxp savecore: reboot after panic: page fault
...
May 14 21:31:36 ldxp syslogd: kernel boot file is
/boot/kernel/kernel
May 14 21:31:36 ldxp savecore: reboot after panic: page fault
...
May 18 09:10:37 ldxp syslogd: kernel boot file is
/boot/kernel/kernel
May 18 09:10:37 ldxp savecore: reboot after panic: page fault



My first thought was RAM so I did a memtest86 over night but
everthing looks ok.


I was able to get and open a kernel dump but have no idea what
this stuff means :)
%sudo cat /var/crash/info.14 
Dump header from device /dev/ar0s1b
  Architecture: i386
  Architecture Version: 16777216
  Dump Length: 1073717248B (1023 MB)
  Blocksize: 512
  Dumptime: Sun May 18 09:03:50 2008
  Hostname: ldxp.xxx.zz
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 5.4-STABLE #5: Mon Jan 14 11:21:51 CET
2008
    pmaechler at ldxp.xxx.zz:/usr/obj/usr/src/sys/LDXP
  Panic String: page fault
  Dump Parity: 3477207326
  Bounds: 14
  Dump Status: good
%

%sudo kgdb kernel.debug /var/crash/vmcore.14
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License,
and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty"
for details.
This GDB was configured as "i386-marcel-freebsd".
#0  doadump () at pcpu.h:160
160             __asm __volatile("movl %%fs:0,%0" : "=r" (td));

(kgdb) bt
#0  doadump () at pcpu.h:160
#1  0xc04c5ae8 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:412
#2  0xc04c5d7c in panic (fmt=0xc06279fc "%s") at
/usr/src/sys/kern/kern_shutdown.c:568
#3  0xc0604c0c in trap_fatal (frame=0xe4aacc24, eva=36) at
/usr/src/sys/i386/i386/trap.c:817
#4  0xc06043f9 in trap (frame=
      {tf_fs = -458620904, tf_es = -1068695536, tf_ds = 16,
tf_edi = -1041254912, tf_esi = -1041254912, tf_ebp = -458568584,
tf_isp = -458568624, tf_ebx = -1035129472, tf_edx = -1035799228,
tf_ecx = -1066880192, tf_eax = 0, tf_trapno = 12, tf_err = 0,
tf_eip = -1068615581, tf_cs = 8, tf_eflags = 65683, tf_esp = 68,
tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:255
#5  0xc05f4e4a in calltrap () at
/usr/src/sys/i386/i386/exception.s:140
#6  0xe4aa0018 in ?? ()
#7  0xc04d0010 in thread_single (mode=-1041254912) at
/usr/src/sys/kern/kern_thread.c:812
#8  0xc04e3ede in turnstile_wait (ts=0xc242f5c0, lock=0xc068b340,
owner=0xc24d2d80) at /usr/src/sys/kern/subr_turnstile.c:556
#9  0xc04bd499 in _mtx_lock_sleep (m=0xc068b340, td=0xc1efb600,
opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:552
#10 0xc04cbf8a in msleep (ident=0xc068b7e4, mtx=0xc068b340,
priority=68, wmesg=0xc0635a53 "psleep", timo=500)
    at /usr/src/sys/kern/kern_synch.c:239
#11 0xc05cee9c in vm_pageout () at
/usr/src/sys/vm/vm_pageout.c:1529
#12 0xc04b14d0 in fork_exit (callout=0xc05cec1c <vm_pageout>,
arg=0x0, frame=0xe4aacd38) at /usr/src/sys/kern/kern_fork.c:791
#13 0xc05f4eac in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:209
(kgdb)


Can anyone give me a hint where else to look for the problem?

tia

Philippe