panic: Exit: Single threading fouled up
Daniel Eischen
eischen at vigrid.com
Mon Apr 26 14:20:01 PDT 2004
On Mon, 26 Apr 2004, Julian Elischer wrote:
>
> On Mon, 26 Apr 2004, Dan Nelson wrote:
>
> > In the last episode (Apr 26), Gavin Atkinson said:
> > > I've seen this panic twice now, once on a heavily loaded UP machine
> > > running gnome at the time, and once on an SMP (hyperthreaded) machine
> > > which was mostly idle as it was shutting down. Both running with ULE.
> >
> > I've gotten it 6 times while running the pike testsuite, but not
> > reliably enough that I can run a WITNESS kernel for a couple hours and
> > catch it. SMP system, 4BSD scheduler, libpthread. Hangs trying to
> > flush buffers so it has never generated a crashdump. The couple of
> > times I was able to break into the debugger before the hang, a ps
> > showed most of the processes in the system waiting for the "proctree"
> > mutex.
>
> By chance I'm reading around that code at the moment..
> here's what is going on..
>
> When a threaded process exits, all threads except the one that is
> actually doing the exit() are forced to abort.
> THEORETICALLY the first thread to get to this code
> should run thread_single() and if another thread calls exit() it should
> block looking for the proc lock until the first thread has successfully
> set the "die-you-scum" flag and then proceed on and see that flag and
> just commit suicide.
>
> actual code is (simplified):
>
> PROC_LOCK(p);
> if (p->p_flag & P_SA || p->p_numthreads > 1) {
> thread_suspend_check(0);
> if (thread_single(SINGLE_EXIT))
> panic ("Exit: Single threading fouled up");
> /*
> where
> thread_single(SINGLE_EXIT)
> is: (simplified)
>
> thread_single(int force_exit)
> {
> struct proc *p;
>
> td = curthread;
> p = td->td_proc;
>
> if ((p->p_flag & P_SA) == 0 && p->p_numthreads == 1)
> return (0);
>
> /* Is someone already single threading? */
> if (p->p_singlethread)
> return (1);
>
> if (force_exit == SINGLE_EXIT) {
> p->p_flag |= P_SINGLE_EXIT;
> } else
> p->p_flag &= ~P_SINGLE_EXIT;
> p->p_flag |= P_STOPPED_SINGLE;
> mtx_lock_spin(&sched_lock);
> p->p_singlethread = td;
> [...]
> [set flags that should trigger thread_suspend_check()]
> return (0);
> }
>
> This means that, despite the fact that the proc lock is required,
> two threads have managed to get into the same code.
>
> the thread_suspend_check(0); for the 2nd thread coming in should just
> abort the thread and never return, so it should never proceed on to see
> the p->p_singlethread already set to non-NULL.
>
> but it does.. hence this panic..
>
> possibly something is setting it earlier
> (There are other times we force single-threading)
> and forgetting to unset it.
> (e.g. in fork or exec)
>
> it is unset using the thread_single_end() call.
There is a race in kse_release() and kse_wakeup() that looks
to have been introduced when sleepqueues were added. msleep()
drops the mutex too early (in this case the proc lock) and
TDF_SINTR gets set after the mutex has been released. So
the code in thread_single():
if (force_exit == SINGLE_EXIT) {
if (TD_IS_SUSPENDED(td2)) {
thread_unsuspend_one(td2);
}
-> if (TD_ON_SLEEPQ(td2) &&
-> (td2->td_flags & TDF_SINTR)) {
sleepq_abort(td2);
}
may not abort the thread in the sleep queue.
--
Dan Eischen
More information about the freebsd-current
mailing list