need for another mutex type/flag?

[David Schultz]

On Sun, Jan 25, 2009, Julian Elischer wrote:
> Jeff Roberson wrote:
> >What is the case you have where you can not acquire non leaf mutexes?
> >
> An example is netgraph. In netgraph you have one (or a small number) 
> thread which is processing on behalf of a number of nodes which may be
> doing work on behalf of completely different entities.
> If such a worker thread blocks then other work becomes starved..

It sounds like acquiring non-leaf mutexes here is a red herring;
the real issue is that you don't want worker threads to block for
long periods of time. Adding an arbitrary rule about mutexes
doesn't seem like the right approach: it isn't needed for
correctness, it breaks modularity (because you can only call
routines if you can guarantee that they follow the rule;
cf. Robert's comments), and it may need to be violated on
occasion. Tools like dtrace are great for tracking down
performance problems associated with lock contention, and they
don't require adding unnecessary magic to the locking API.

On a related note, one thing that might help performance is to add
a way to create dynamically-sized worker thread pools. This
doesn't help when you have high contention for the same resource
(as all the threads block on the same mutex), but it does help
when you have contention for multiple resources, and several
threads may be blocked at any given time. .NET's ThreadPool and
many commercial application servers have APIs to do things like
this, and kern/vfs_aio.c has a special-purpose thread pool to
handle blocking I/O. Note that I'm not familiar with the guts of
netgraph, so you should take this with a grain of salt and come up
with your own opinion about this idea.