Is MTX_CONTESTED evil?

[David Schultz]

On Mon, Mar 22, 2004, Seigo Tanimura wrote:
> On Tue, 16 Mar 2004 10:09:48 -0500,
>   John Baldwin <john at baldwin.cx> said:
> 
> john> On Tuesday 16 March 2004 12:19 am, Seigo Tanimura wrote:
> >> _mtx_unlock_sleep() currently wakes up only one thread being blocked,
> >> and leaves MTX_CONTESTED on a mutex.  According to Solaris Internals,
> >> that strategy adds an overhead to check for MTX_CONTESTED on a mutex,
> >> even though it is not held by any thread.  The thread waken up cannot
> >> grab the mutex immediately by _obtain_lock() and have to go through
> >> _mtx_lock_sleep().  The penalty tends to be large for a mutex with a
> >> high contention, and we have at least one of such a mutex - Giant.
> >> 
> >> What would it be like if we axed MTX_CONTEST and let
> >> _mtx_unlock_sleep() wake up all of the blocked threads?
> 
> john> We wouldn't be able to axe MTX_CONTEST.  We also use it to determine on unlock 
> john> if we can unlock easily or if we have waiters that we need to awake.  The 
> john> only way we might be able to axe MTX_CONTEST would be to penalize every 
> john> unlock operation requiring a turnstile lookup (spin lock acquire/release + 
> john> hash table lookup) even unlocks of an uncontested mutex.  However, what I 
> john> think you want to do is get rid of the mtx_lock == MTX_CONTESTED case and use 
> john> turnstile_wakeup() rather than turnstile_signal()?  Is that what you are 
> 
> Yes.  What I an wondering is whether the reduction of the cost due to
> a mutex with waiters and no holders can beat the cost of waking up all
> the waiters on the turnstile.

By the way, one thing to keep in mind is that Solaris has working
adaptive mutexes.  For adaptive mutexes, the waiting case is
almost never supposed to happen, so it's more reasonable for them
to wake all waiters.  However, AFAIK, FreeBSD's adaptive mutex
support is incomplete or broken at this point, so you may run into
a thundering herd problem if you wake all waiters.