[RFC] Understanding the locking of struct buf

[Konstantin Belousov]

On Fri, Mar 12, 2021 at 07:57:54PM +0100, Alexander Lochmann wrote:
> Hi folks!
> 
> According to the definition [1], any access to a struct buf is protected
> by a lock:
> 1) Are there exceptions to those locking rules for reading and/or
> writing a member?
> E.g. any read is permitted without a lock being held.
> Can b_bcount, for example, be read without a lock?
Sure you can read it without lock.  Question is, what do you intent to do
with this information.

Buffers used to track io requests for vnodes, that is, buffers appearing on
bufobj dirty/clean queues, are type-stable, so you are guaranteed that the
memory is not reused for something else.  On the other hand, only buffer
lock guarantees that the buffer identity, i.e. (vnode, range) for data
described by the buffer, is valid.

> 
> 2) In vfs_bio.c, for example, the BUF_KERNPROC macro is used to move
> ownership of a buf.b_lock to the kernel.
> In _lockmgr_disown() [2] WITNESS_UNLOCK() is called to emulate an
> unlock. So, shouldn't be there a WITNESS_LOCK() for that lock?
> Otherwise, the Witness system might complain about the upcoming unlock.
Are you reporting a bug or just asking about LK_KERNPROC. Lockmgr
(kern_lock.c)is careful enough to handle LK_KERNPROC owner specially. In
particular, it does not report unlock of such lock to witness.

> 
> 3) Is the function brelse()/bqrelse() considered to be a destruction
> function?
As I said above, buffers are type-stable, they are not destructed.
brelse/bqrelse relinguish ownership of the buffer by the calling thread.
After that, buffer typically goes to corresponding queue of unlocked
clean or dirty buffer, from where it could be picked up or reclaimed to
represent a different buffer.

> Hence, no lock (buf.b_lock) is needed within this function to access a
> struct buf.
The buffer lock is required for call to brelse (and bqrelse). As noted,
buffer lock represents an ownership of the buffer. It is typically
obtained when the thread calls bread(9), which returns valid and locked
buffer. Then caller do whatever he needed, then it is either brelse() if
the buffer is clean, or some variant of bwrite() if buffer become dirty.
Buffer lock is taken somewhere in bread() and released in brelse/bwrite,
from the PoV of the consumer of the buffer cache.

Actual interactions are usually more complex, due to a lot of features
grown in the buffer cache for 40+ years, but the main principle remained
the same, thread that owns the buffer owns its lock.