rwlocks, correctness over speed.

[Attilio Rao]

2007/11/24, Attilio Rao <attilio at freebsd.org>:
> 2007/11/24, Robert Watson <rwatson at freebsd.org>:
> >
> > On Fri, 23 Nov 2007, Stephan Uphoff wrote:
> >
> > >>> I talked with Attilio about that on IRC. Most common cases of writer
> > >>> starvation (but not all) could be solved by keeping a per thread count of
> > >>> shared acquired rwlocks. If a rwlock is currently locked as shared/read
> > >>> AND a thread is blocked on it to lock it exclusively/write - then new
> > >>> shared/read locks will only be granted to thread that already has a shared
> > >>> lock. (per thread shared counter is non zero)
> > >>>
> > >>> To be honest I am a bit twitchy about a lock without priority propagation
> > >>> - especially since in FreeBSD threads run with user priority in kernel
> > >>> space and can get preempted.
> > >>
> > >> That's an interesting hack, I guess it could be done.
> > >>
> > >> I would still like to disallow recursion.
> > >>
> > > Oh - I am all for disallowing recursion. In my opinion the only valid place
> > > for a thread to acquire the same lock multiple times is inside a transaction
> > > system with full deadlock detection. The question is if we can do that this
> > > late in the game? Maybe we could make non recursive the default and add a
> > > call rw_allow_recurse or rw_init_recurse to allow recursion on a lock if we
> > > can't get away with the straight out removal of the option? (Or less
> > > desirable - keep the current default and add functions to disallow
> > > recursion)
> >
> > While I'm no great fan of recursion, the reality is that many of our kernel
> > subsystems are not yet ready to disallow recursion on locks.  Take a look at
> > the cases where we explicitly enable recursive acquisition for mutexes--in
> > practice, most network stack mutexes are recursive due to the recursive
> > calling in the network stack.  While someday I'd like to think we'll be able
> > to eliminate some of that, but it won't be soon since it requires significant
> > reworking of very complicated code.  The current model in which recursion is
> > explicitly enabled only where still required seems to work pretty well for the
> > existing code, although it's hard to say yet in the code I've looked at
> > whether read recursion would be required--the situations I have in mind would
> > require purely write recursion.  There's one case in the UNIX domain socket
> > code where we do a locked test and conditional lock/unlock with an rwlock for
> > exclusive locking because recursion isn't currently supported, and that's not
> > a usage I'd like to encourage more of.
>
> I think however that Stephan is just referring to the readers
> recursion (as we are doing) that if present should be just in few
> points.
> If we want todo a radical switch of this genre this is the right
> moment, just before rwlock became too widespread.
>
> I have a patch against witness which would check for readers recursion
> and panic, I will post in the night or tomorrow.

So, this patch checks for recursive readers and will panic in case:
http://people.freebsd.org/~attilio/recursion_checks.diff

(I preferred to do in this way, but you can check for perforce commit
number 129792 for an alternative way to achieve the same).
What I'm looking for now is for people which can apply this patch,
enable WITNESS with all the relevant options on and try to see if
there are recursive readers in rwlocks somewhere (hopefully not after
the pfil(9) switching to rmlock that mlaier did recently).

Attilio


-- 
Peace can only be achieved by understanding - A. Einstein