location of bioq lock

[Robert Watson]

On Fri, 8 Jul 2005, Luigi Rizzo wrote:

> 1) put the lock in the struct bio_queue_head.
>   This is the same thing done in struct g_bioq defined in
>   sys/geom/geom.h . Quite clean, except that perhaps some
>   users of bio_queue_head still run under Giant (e.g. cam/scsi ?)
>   and so it is not possible to 'bring in' the lock.
>
> 2) change bioq_init() so that it takes also a pointer to the mtx
>   that protects the queue.
>   This is probably less clean, but perhaps a bit more flexible because
>   the queue and its lock are decoupled. Also it permits to deal
>   with the 'Giant' case easily.
>
> Other ideas ?

In the network stack work, we started out with locks tightly coupled with 
the queues they protected as part of an early design decision to embed 
mutexes in ifqueue's, one of the widely used queueing structures.  We're 
actually exploring backing off that decision now such that components use 
queues as a "library", and use their own synchronization to protect the 
queue.  This allows, for example, lock coalescing across multiple queues, 
or combining of queue mutexes with larger component locks.  It also allows 
queues to be agnostic of the lock type that is used to protect them, so a 
consumer could use an ex/rwlock or the like, or for that matter a spin 
lock.  It also allows lock-free access to the queue where that is 
appropriate.

FYI, one interesting point regarding lock order: in the ifqueue lock 
model, ifqueue locks were leaf mutexes.  However, if the scope of locks 
protection queues expands, the replacement locks may well not be leaf 
mutexes.  This is relevant in "hand-off" scenarios, where before 
contention on a queue mutex was very unlikely during a "grab, insert, 
drop" scenario, contention chances are increased as the lock might also 
cover other significantly time-consuming things, such as data copies, 
hardware I/O interactions, and so on.  This might, or might not, outweight 
the overhead of the additional locking, and is worth keeping in mind.

So based on that experience, my suggestion is to make locking a property 
of the consumer of the API, not the provider, and to create macros or 
queue wrapper functions in the consumer that *are* aware of the locking 
semantics, for the purposes of code simplification, assertion placement, 
and so on.

However, the network stack is fairly different from the storage I/O stack, 
so the lessons (while interesting) might well not hold there.

Robert N M Watson