RELENG_6 panic under heavy load

[Gleb Smirnoff]

  I wonder why UMA was suspected to be the problem. Dima gave
me access to the core. Here are more details from the trace:

Unread portion of the kernel message buffer:
panic: thread 100147(nagios):1 holds process lock but isn't blocked on a lock

#9  0xd060038e in panic (fmt=0xd08094d9 "thread %d(%s):%d holds %s but isn't blocked on a lock\n") at /usr/src/sys/kern/kern_shutdown.c:549
#10 0xd0629228 in propagate_priority (td=0xd745c900) at /usr/src/sys/kern/subr_turnstile.c:239
#11 0xd0629f32 in turnstile_wait (lock=0xd5dd5498, owner=0xd745c900) at /usr/src/sys/kern/subr_turnstile.c:643
#12 0xd05f4fc1 in _mtx_lock_sleep (m=0xd5dd5498, tid=3583683968, opts=0, file=0x12 <Address 0x12 out of bounds>, line=18) at /usr/src/sys/kern/kern_mutex.c:579
#13 0xd05f4992 in _mtx_lock_flags (m=0xd5dd5498, opts=0, file=0xd0806c3d "/usr/src/sys/kern/kern_thread.c", line=824) at /usr/src/sys/kern/kern_mutex.c:288
#14 0xd060d340 in thread_single (mode=0) at /usr/src/sys/kern/kern_thread.c:824
#15 0xd05e38b9 in fork1 (td=0xd59aad80, flags=20, pages=0, procp=0xf5cacccc) at /usr/src/sys/kern/kern_fork.c:274
#16 0xd05e3509 in fork (td=0xd59aad80, uap=0xf5cacd04) at /usr/src/sys/kern/kern_fork.c:98
#17 0xd07a6d10 in syscall (frame=
      {tf_fs = 134938683, tf_es = 59, tf_ds = -809566149, tf_edi = 134953856, tf_esi = 673312612, tf_ebp = -809526568, tf_isp = -171258524, tf_ebx = 672261300, tf_edx = 0, tf_ecx = 134963456, tf_eax = 2, tf_trapno = 12, tf_err = 2, tf_eip = 672684403, tf_cs = 51, tf_eflags = 642, tf_esp = -809526660, tf_ss = 59})
    at /usr/src/sys/i386/i386/trap.c:983
#18 0xd078f38f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#19 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) frame 10
#10 0xd0629228 in propagate_priority (td=0xd745c900) at /usr/src/sys/kern/subr_turnstile.c:239
239                     KASSERT(TD_ON_LOCK(td), (
(kgdb) list
234     #endif
235
236                     /*
237                      * If we aren't blocked on a lock, we should be.
238                      */
239                     KASSERT(TD_ON_LOCK(td), (
240                         "thread %d(%s):%d holds %s but isn't blocked on a lock\n",
241                         td->td_tid, td->td_proc->p_comm, td->td_state,
242                         ts->ts_lockobj->lo_name));
243
(kgdb) frame 14
#14 0xd060d340 in thread_single (mode=0) at /usr/src/sys/kern/kern_thread.c:824
824                     PROC_LOCK(p);
(kgdb) list
819                     thread_stopped(p);
820                     thread_suspend_one(td);
821                     PROC_UNLOCK(p);
822                     mi_switch(SW_VOL, NULL);
823                     mtx_unlock_spin(&sched_lock);
824                     PROC_LOCK(p);
825                     mtx_lock_spin(&sched_lock);
826                     if (mode == SINGLE_EXIT)
827                             remaining = p->p_numthreads;
828                     else if (mode == SINGLE_BOUNDARY)

-- 
Totus tuus, Glebius.
GLEBIUS-RIPN GLEB-RIPE