kthread_exit(9) unexpectedness

[John Baldwin]

On Thursday 20 November 2008 05:22:03 pm Lawrence Stewart wrote:
> John Baldwin wrote:
> > On Wednesday 19 November 2008 08:21:44 am Lawrence Stewart wrote:
> >> Hi all,
> >>
> >> I tracked down a deadlock in some of my code today to some weird 
> >> behaviour in the kthread(9) KPI. The executive summary is that 
> >> kthread_exit() thread termination notification using wakeup() behaves as 
> >> expected intuitively in 8.x, but not in 7.x.
> > 
> > In 5.x/6.x/7.x kthreads are still processes and it has always been a 
wakeup on 
> > the proc pointer.  kthread_create() in 7.x returns a proc pointer, not a 
> > thread pointer for example.  In 8.x kthreads are actual threads and 
> 
> Yep, but the processes have a *thread in them right? The API naming is 
> obviously slightly misleading, but it essentially creates a new single 
> threaded process prior to 8.x.

Yes, but you have to go explicitly use FIRST_THREAD_IN_PROC().  Most of the 
kernel modules I've written that use kthread's in < 8 do this:

static struct proc *foo_thread;

/* Called for MOD_LOAD. */
static void
load(...)
{

	error = kthread_create(..., &foo_thread);
}

static void
unload(...)
{

	/* set flag */
	msleep(foo_thread, ...);
}

And never actually use the thread at all.  However, if you write the code for 
8.x, now you _do_ get a kthread and sleep on the thread so it becomes:

static struct thread *foo_thread;

static void
load(...)
{

	error = kproc_kthread_add(..., proc0, &foo_thread);
}

static void
unload(...)
{

	/* set flag */
	msleep(foo_thread, ...);
}

> > kthread_add() and kproc_kthread_add() both return thread pointers.  Hence 
in
> 
> Yup.
> 
> > 8.x kthread_exit() is used for exiting kernel threads and wakes up the 
thread 
> > pointer, but in 7.x kthread_exit() is used for exiting kernel processes 
and 
> > wakes up the proc pointer.  I think what is probably needed is to simply 
> 
> In the code, yes. Our documented behaviour as far as I can tell is 
> different though, unless we equate a "thread handle" to "proc handle" 
> prior to 8.x, which I don't think is the case -  they are still different.

It has always been the case in < 8 that you sleep on the proc handle (what 
kthread_create() actually returns in < 8).  And in fact, you even have to dig 
around in the proc you get from kthread_create() to even find the thread 
pointer as opposed to having the API hand it to you.

> > document that arrangement as such.  Note that the sleeping on proc pointer 
> 
> I agree that the arrangement needs to be better documented. The change 
> in 8.x is subtle enough that reading the kthread man page in 7.x and 8.x 
> doesn't immediately make it obvious what's going on.
> 
> > has been the documented way to synchronize with kthread_exit() since 5.0.
> > 
> 
> Could you please point me at this documentation? I've missed it in my 
> poking around thus far.

It is probably only documented in numerous threads in the mail archives sadly, 
but there have been several of them and there have been several fixes to get 
this right (the randomdev thread and fdc(4) thread come to mind).

-- 
John Baldwin