Unexplained kernel panic on 5-STABLE (now in 6-STABLE)

[Peter van Heusden]

Thanks for the advice John. I upgraded to 6-STABLE and just got a kernel
panic again. Before I list the dump, I'd like to mention two messages I
see in my syslog. Firstly, often I get something like this:

kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 151698,
size: 28672

(though the last message like this was at 9:18 this morning and the
kernel paniced at about 10:04)

and secondly, during boot, I get messages like this:

kernel: acpi: bad read from port 0xcfc (32)
kernel: acpi: bad write to port 0xcf8 (32), val 0x80002084

Anyway, on with the kgdb output:

leftside# kgdb kernel.debug /var/crash/vmcore.27
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x1b
fault code              = supervisor write, page not present
instruction pointer     = 0x20:0xc08a98ca
stack pointer           = 0x28:0xcbdd4cc4
frame pointer           = 0x28:0xcbdd4ce0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 34 (pagedaemon)
trap number             = 12
panic: page fault
Uptime: 19h6m20s
Dumping 251 MB (2 chunks)
  chunk 0: 1MB (160 pages) ... ok
  chunk 1: 251MB (64252 pages) 236 220 204 188 172 156 140 124 108 92 76
60 44 28 12

#0  doadump () at pcpu.h:165
165             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) list *0xc08a98ca
0xc08a98ca is in pmap_ts_referenced (atomic.h:149).
144     static __inline int
145     atomic_cmpset_int(volatile u_int *dst, u_int exp, u_int src)
146     {
147             int res = exp;
148    
149             __asm __volatile (
150             "       " __XSTRING(MPLOCKED) " "
151             "       cmpxchgl %2,%1 ;        "
152             "       setz    %%al ;          "
153             "       movzbl  %%al,%0 ;       "
(kgdb) backtrace
#0  doadump () at pcpu.h:165
#1  0xc069cee6 in boot (howto=260) at
/usr/freebsd6/src/sys/kern/kern_shutdown.c:409
#2  0xc069d17c in panic (fmt=0xc0903b6f "%s") at
/usr/freebsd6/src/sys/kern/kern_shutdown.c:565
#3  0xc08ac6d4 in trap_fatal (frame=0xcbdd4c84, eva=27) at
/usr/freebsd6/src/sys/i386/i386/trap.c:836
#4  0xc08ac43b in trap_pfault (frame=0xcbdd4c84, usermode=0, eva=27) at
/usr/freebsd6/src/sys/i386/i386/trap.c:744
#5  0xc08ac079 in trap (frame=
      {tf_fs = -874708984, tf_es = -1064697816, tf_ds = -1063452632,
tf_edi = -1054911176, tf_esi = 4, tf_ebp = -874689312, tf_isp =
-874689360, tf_ebx = -1050447872, tf_edx = -1, tf_ecx = -1038927232,
tf_eax = 4, tf_trapno = 12, tf_err = 2, tf_eip = -1064658742, tf_cs =
32, tf_eflags = 66050, tf_esp = -874689320, tf_ss = 0}) at
/usr/freebsd6/src/sys/i386/i386/trap.c:434
#6  0xc089a7fa in calltrap () at
/usr/freebsd6/src/sys/i386/i386/exception.s:139
#7  0xc08a98ca in pmap_ts_referenced (m=0xc11f5538) at atomic.h:149
#8  0xc0815e59 in vm_pageout_page_stats () at
/usr/freebsd6/src/sys/vm/vm_pageout.c:1401
#9  0xc0816192 in vm_pageout () at
/usr/freebsd6/src/sys/vm/vm_pageout.c:1546
#10 0xc0687434 in fork_exit (callout=0xc0815ef8 <vm_pageout>, arg=0x0,
frame=0xcbdd4d38) at /usr/freebsd6/src/sys/kern/kern_fork.c:805
#11 0xc089a85c in fork_trampoline () at
/usr/freebsd6/src/sys/i386/i386/exception.s:208

Thanks for all the help,
Peter

John Baldwin wrote:
> On Monday 14 August 2006 14:16, Peter van Heusden wrote:
>   
>> Thanks. That gives the following output:
>>
>> #9  0xc0801295 in trap_pfault (frame=0xd1231b68, usermode=0x0, eva=0x3)
>> at /usr/src/sys/i386/i386/trap.c:714
>> #10 0xc0800fa5 in trap (frame=
>>       {tf_fs = 0xc1e20018, tf_es = 0x10, tf_ds = 0x10, tf_edi = 0x0,
>> tf_esi = 0xc1045420, tf_ebp = 0xd1231bb8, tf_isp = 0xd1231b94, tf_ebx =
>> 0xc1045458, tf_edx = 0xffffffff, tf_ecx = 0xc28cf000, tf_eax =
>> 0xc1045434, tf_trapno = 0xc, tf_err = 0x2, tf_eip = 0xc07b6bbe, tf_cs =
>> 0x8, tf_eflags = 0x10286, tf_esp = 0xc102dd08, tf_ss = 0xc1edf570})
>>     at /usr/src/sys/i386/i386/trap.c:427
>> #11 0xc07f0eea in calltrap () at /usr/src/sys/i386/i386/exception.s:140
>> ...
>> #25 0xc07b6bbe in uma_zalloc_arg (zone=0xc1045420, udata=0x0, flags=0x1)
>> at /usr/src/sys/vm/uma_core.c:1895
>> #26 0xc07fc97f in get_pv_entry () at uma.h:276
>>     
>
> So it got a nested page fault inside the VM, basically.
>
>   
>> Does this help? It seems that fork() was called, and then something went
>> wrong from there. One common feature of these panics seems to be that
>> they happen when my server (an aging P3 700Mhz with 256 MB of RAM that
>> is put to use for all sorts of network services) is under quite heavy load.
>>     
>
> To be honest, I'd update it to 6.1 or even 6-stable as this is likely already 
> fixed in 6.x.
>
>