Why do we need to acquire the current thread's lock before context switching?

[John Baldwin]

On Thursday, September 12, 2013 4:00:56 pm Dheeraj Kandula wrote:
> Hey John,
>        I think I get it now clearly.
> 
> The td_lock of each thread actually points to the Thread Queue's lock on
> which it is present. i.e. run queue which may either be the real time runq,
> timeshare runq or the idle runq. For sleep the td_lock points to the
> blocked_lock which is a global lock protecting the sleep queue I think.
> 
> Before cpu_switch() is invoked, the old thread's td_lock is released as
> shown below: the code is from sched_switch of sched_ule.c
> 
>  lock_profile_release_lock<http://nxr.netbsd.org/source/s?defs=lock_profile_release_lock&project=src-freebsd>
> (&TDQ_LOCKPTR<http://nxr.netbsd.org/source/xref/src-freebsd/sys/kern/sched_ule.c#TDQ_LOCKPTR>
> (tdq)->lock_object<http://nxr.netbsd.org/source/s?defs=lock_object&project=src-freebsd>
> );
> 
> TDQ_LOCKPTR <http://nxr.netbsd.org/source/xref/src-freebsd/sys/kern/sched_ule.c#TDQ_LOCKPTR>(tdq)->mtx_lock
> <http://nxr.netbsd.org/source/s?defs=mtx_lock&project=src-freebsd> =
> (uintptr_t <http://nxr.netbsd.org/source/s?defs=uintptr_t&project=src-freebsd>)newtd
> <http://nxr.netbsd.org/source/xref/src-freebsd/sys/kern/sched_ule.c#newtd>;

This is not releasing the td_lock of the old thread.  TDQ_LOCK() is
td_lock of the new thread that is about to run, and the assignment
to mtx_lock is transferring ownership of TDQ_LOCK() from the old thread
to the new thread.  The lock_profile calls fake an unlock / lock so the
profiling code doesn't get confused by the handoff, but TDQ_LOCK() is
not unlocked here.

The old thread's td_lock is left alone if it is already TDQ_LOCK()
(notice that in this snippet, the various branches assert this to be true):

	/*
	 * The lock pointer in an idle thread should never change.  Reset it
	 * to CAN_RUN as well.
	 */
	if (TD_IS_IDLETHREAD(td)) {
		MPASS(td->td_lock == TDQ_LOCKPTR(tdq));
		TD_SET_CAN_RUN(td);
	} else if (TD_IS_RUNNING(td)) {
		MPASS(td->td_lock == TDQ_LOCKPTR(tdq));
		...

However, in the other cases, this code is called:

	} else {
		/* This thread must be going to sleep. */
		TDQ_LOCK(tdq);
		mtx = thread_lock_block(td);
		tdq_load_rem(tdq, td);
	}

thread_lock_block() changes td_lock in the old thread to point to a
dummy lock called the "block_lock" and then unlocks the old thread's
td_lock.  However, it returns the previous value of td_lock as 'mtx'.
That is then passed to cpu_switch().  cpu_switch() restores td_lock
in the old thread to 'mtx'.

The effect of this is to temporarily assign td_lock of the old thread
to block_lock while the thread finishes switching out, but to be able
to release the old thread's associated td_lock in C rather than having
to do it from cpu_switch().  The 'block_lock' is a special lock that
is always locked and never unlocked.  That causes any other threads
that are trying to lock the old thread to spin until the old thread
is finished switching out even after the old thread's td_lock has
been released (since the new thread will spin on block_lock until
cpu_switch() restores td_lock).
 
> Later after cpu_switch is done,
> 
> 
> lock_profile_obtain_lock_success
> <http://nxr.netbsd.org/source/s?defs=lock_profile_obtain_lock_success&project=src-freebsd>(&TDQ_LOCKPTR
> <http://nxr.netbsd.org/source/xref/src-freebsd/sys/kern/sched_ule.c#TDQ_LOCKPTR>(tdq)->lock_object
> <http://nxr.netbsd.org/source/s?defs=lock_object&project=src-freebsd>,
> 0, 0, __FILE__ <http://nxr.netbsd.org/source/s?defs=__FILE__&project=src-freebsd>,
> __LINE__ <http://nxr.netbsd.org/source/s?defs=__LINE__&project=src-freebsd>);
> 
> 
> is executed which locks the lock of the thread queue on the current
> CPU which can be on a different CPU. I assume the new thread's td_lock
> points to the current CPU's thread queue.

As with the first hunk, this is not actually locking the lock, just
pacifying the profiling code.  The new thread's td_lock does indeed
point to the current CPU's thread queue.  This is known to be true
because the new thread was chosen from the current CPU's thread
queue.

> Now it is clear that the mutex is unlocked by the same thread that locks it.

Except not in this one case. :)  In the case of a context switch, the
old thread locks TDQ_LOCK(), sets td_lock to blocked_lock and drops
its old td_lock, and changes the internals of TDQ_LOCK() so that it is
now owned by the new thread.  cpu_switch() is called which then restores
td_lock in the old thread, switches the MD context (stack and registers,
etc.).  The new thread returns on its own stack, and it now owns the
TDQ_LOCK() that was locked by the old thread.  However, since the
ownership was transferred, it can unlock TDQ_LOCK().

Note that if a running thread is preempted, it doesn't do the block_lock
business, instead it just transfers ownership of the TDQ_LOCK() it
already holds to the new thread and never explicitly unlocks that lock.

-- 
John Baldwin