Unexplained kernel panic on 5-STABLE

Mon Aug 14 18:17:34 UTC 2006

Thanks. That gives the following output:

243                     ts = td->td_blocked;
(kgdb) p td->td_proc->p_pid
$1 = 0x126ce
(kgdb) proc 0x126ce
(kgdb) backtrace
#0  0xc065fe6b in sched_switch (td=0xc1e22c00, newtd=0xc14edc00,
flags=0x1) at /usr/src/sys/kern/sched_4bsd.c:881
#1  0xc0656866 in mi_switch (flags=0x1, newtd=0x0) at
/usr/src/sys/kern/kern_synch.c:355
#2  0xc066c72a in sleepq_switch (wchan=0x0) at
/usr/src/sys/kern/subr_sleepqueue.c:406
#3  0xc066c913 in sleepq_wait (wchan=0xc1edf2cc) at
/usr/src/sys/kern/subr_sleepqueue.c:518
#4  0xc062838b in cv_wait (cvp=0xc1edf2cc, mp=0xc09232d8) at
/usr/src/sys/kern/kern_condvar.c:128
#5  0xc0655f04 in _sx_xlock (sx=0xc1edf29c, file=0x0, line=0x0) at
/usr/src/sys/kern/kern_sx.c:175
#6  0xc07a73fb in _vm_map_lock_read (map=0x0, file=0x0, line=0x0) at
/usr/src/sys/vm/vm_map.c:380
#7  0xc07aa53a in vm_map_lookup (var_map=0xd1231aa4, vaddr=0x0,
fault_typea=0x2, out_entry=0xd1231aa8, object=0x0,
    pindex=0x0, out_prot=0x0, wired=0xd1231a80) at
/usr/src/sys/vm/vm_map.c:2998
#8  0xc07a2e99 in vm_fault (map=0xc1edf258, vaddr=0x0, fault_type=0x2,
fault_flags=0x8)
    at /usr/src/sys/vm/vm_fault.c:229
#9  0xc0801295 in trap_pfault (frame=0xd1231b68, usermode=0x0, eva=0x3)
at /usr/src/sys/i386/i386/trap.c:714
#10 0xc0800fa5 in trap (frame=
      {tf_fs = 0xc1e20018, tf_es = 0x10, tf_ds = 0x10, tf_edi = 0x0,
tf_esi = 0xc1045420, tf_ebp = 0xd1231bb8, tf_isp = 0xd1231b94, tf_ebx =
0xc1045458, tf_edx = 0xffffffff, tf_ecx = 0xc28cf000, tf_eax =
0xc1045434, tf_trapno = 0xc, tf_err = 0x2, tf_eip = 0xc07b6bbe, tf_cs =
0x8, tf_eflags = 0x10286, tf_esp = 0xc102dd08, tf_ss = 0xc1edf570})
    at /usr/src/sys/i386/i386/trap.c:427
#11 0xc07f0eea in calltrap () at /usr/src/sys/i386/i386/exception.s:140
#12 0xc1e20018 in ?? ()
#13 0x00000010 in ?? ()
#14 0x00000010 in ?? ()
#15 0x00000000 in ?? ()
#16 0xc1045420 in ?? ()
#17 0xd1231bb8 in ?? ()
#18 0xd1231b94 in ?? ()
#19 0xc1045458 in ?? ()
#20 0xffffffff in ?? ()
#21 0xc28cf000 in ?? ()
#22 0xc1045434 in ?? ()
#23 0x0000000c in ?? ()
#24 0x00000002 in ?? ()
#25 0xc07b6bbe in uma_zalloc_arg (zone=0xc1045420, udata=0x0, flags=0x1)
at /usr/src/sys/vm/uma_core.c:1895
#26 0xc07fc97f in get_pv_entry () at uma.h:276
#27 0xc07fca74 in pmap_insert_entry (pmap=0xc1edf570, va=0x0,
m=0xc131ce78) at /usr/src/sys/i386/i386/pmap.c:1563
#28 0xc07fdd9e in pmap_copy (dst_pmap=0xc1edf570, src_pmap=0xc1edf318,
dst_addr=0x281ad000, len=0x9000,
    src_addr=0x281b5000) at /usr/src/sys/i386/i386/pmap.c:2409
#29 0xc07a99f4 in vm_map_copy_entry (src_map=0xc1edf258,
dst_map=0xc1edf4b0, src_entry=0xc28ed7f8,
    dst_entry=0xc1cc02a8) at /usr/src/sys/vm/vm_map.c:2426
#30 0xc07a9ce6 in vmspace_fork (vm1=0xc1edf258) at
/usr/src/sys/vm/vm_map.c:2581
#31 0xc07a5a6b in vm_forkproc (td=0xc1e22c00, p2=0xc2648a98,
td2=0xc23ec000, flags=0x14)
    at /usr/src/sys/vm/vm_glue.c:464
#32 0xc063b5cc in fork1 (td=0xc1e22c00, flags=0x14, pages=0x0,
procp=0xd1231cd4) at /usr/src/sys/kern/kern_fork.c:644
#33 0xc063a50c in fork (td=0xc1e22c00, uap=0xd1231d04) at
/usr/src/sys/kern/kern_fork.c:97
#34 0xc0801907 in syscall (frame=
      {tf_fs = 0xc07f002f, tf_es = 0x2f, tf_ds = 0x2f, tf_edi =
0xbfbfeb00, tf_esi = 0xa25d811, tf_ebp = 0xbfbfeac8, tf_isp =
0xd1231d64, tf_ebx = 0x2817599c, tf_edx = 0x0, tf_ecx = 0xbfbfeb00,
tf_eax = 0x2, tf_trapno = 0xc, tf_err = 0x2, tf_eip = 0x28206467, tf_cs
= 0x1f, tf_eflags = 0x206, tf_esp = 0xbfbfeabc, tf_ss = 0x2f})
    at /usr/src/sys/i386/i386/trap.c:1014
#35 0xc07f0f3f in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:201
#36 0xc07f002f in alloc_bounce_zone (dmat=0xbfbfeb00) at
/usr/src/sys/i386/i386/busdma_machdep.c:967

Does this help? It seems that fork() was called, and then something went
wrong from there. One common feature of these panics seems to be that
they happen when my server (an aging P3 700Mhz with 256 MB of RAM that
is put to use for all sorts of network services) is under quite heavy load.

Peter

John Baldwin wrote:
> On Sunday 13 August 2006 03:36, Peter van Heusden wrote:
>   
>> Hi everyone
>>
>> Almost every day, some time around midnight, my FreeBSD 5-STABLE kernel
>> panics. I've attached the output of a kgdb session of the saved core
>> file - is there any further info that would be useful to debugging? I've
>> got no idea where to start looking for a solution to this problem, so
>> please help!!
>>     
>
> A thread is sleeping while holding a mutex.
>
>   
>> (kgdb) list *0xc066dfc3
>> 0xc066dfc3 is in propagate_priority
>> (/usr/src/sys/kern/subr_turnstile.c:245).
>> 240                     /*
>> 241                      * Pick up the lock that td is blocked on.
>> 242                      */
>> 243                     ts = td->td_blocked;
>> 244                     MPASS(ts != NULL);
>> 245                     tc = TC_LOOKUP(ts->ts_lockobj);
>> 246                     mtx_lock_spin(&tc->tc_lock);
>> 247    
>> 248                     /*
>> 249                      * This thread may not be blocked on this
>> turnstile anymore
>> (kgdb) backtrace
>> #19 0xc066dfc3 in propagate_priority (td=0xc1e22c00) at
>> /usr/src/sys/kern/subr_turnstile.c:243
>>     
>
> Do the following in 'kgdb':
>
> frame 19
> p td->td_proc->p_pid
> (this will output the 'pid' of the misbehaving thread)
> proc <pid>
> (<pid> is the pid from the previous command, this switches you to that
>  thread)
> backtrace
>
> That backtrace will then show you which thread slept while holding a
> mutex.
>
>   