LOR in dirhash

John Baldwin jhb at freebsd.org
Tue Feb 1 14:53:16 UTC 2011

On Tuesday, February 01, 2011 4:54:15 am Robert Schulze wrote:
> We are currently having continuous problems with 8-STABLE.
> Since we upgraded, we have "double faults" every three days or so, 
> yesterday we built a debugging kernel to identify the problem.
> Today I saw a LOR in /var/log/messages, the whole output:
> 8<----------------------------------------------------------
> lock order reversal:
> 1st 0xffffff81ef08ce98 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2636
> 2nd 0xffffff00157b4600 dirhash (dirhash) @ 
> /usr/src/sys/ufs/ufs/ufs_dirhash.c:285
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> kdb_backtrace() at kdb_backtrace+0x37
> _witness_debugger() at _witness_debugger+0x49
> witness_checkorder() at witness_checkorder+0x7d3
> _sx_xlock() at _sx_xlock+0x4a
> ufsdirhash_acquire() at ufsdirhash_acquire+0x3a
> ufsdirhash_add() at
> ufsdirhash_add+0x19
> ufs_diren
> ter() at
> ufs_direnter+0x876
> ufs_makeinode() at
> ufs_makeinode+0x239
> vn_open_cred() at vn_open_cred+0x415
> kern_openat() at
> kern_openat+0x165
> syscallenter() at syscallenter+0xe5
> syscall() at
> syscall+0x55
> Xfast_syscall() at
> Xfast_syscall+0xe2
> --- syscall (5, FreeBSD ELF64
> , open)
> , rip =
> 0x8009
> a4a7c,
> rsp = 0x
> 7ffffff
> e918, r
> bp = 0x1 ---
> 8<----------------------------------------------------------
> maybe this is a seriuos one?

From the source code in ufs_dirhash.c:

 * Locking:
 * The relationship between inode and dirhash is protected either by an
 * exclusive vnode lock or the vnode interlock where a shared vnode lock
 * may be used.  The dirhash_mtx is acquired after the dirhash lock.  To
 * handle teardown races, code wishing to lock the dirhash for an inode
 * when using a shared vnode lock must obtain a private reference on the
 * dirhash while holding the vnode interlock.  They can drop it once they
 * have obtained the dirhash lock and verified that the dirhash wasn't
 * recycled while they waited for the dirhash lock.
 * ufsdirhash_build() acquires a shared lock on the dirhash when it is
 * successful.  This lock is released after a call to ufsdirhash_lookup().
 * Functions requiring exclusive access use ufsdirhash_acquire() which may
 * free a dirhash structure that was recycled by ufsdirhash_recycle().
 * The dirhash lock may be held across io operations.
 * WITNESS reports a lock order reversal between the "bufwait" lock
 * and the "dirhash" lock.  However, this specific reversal will not
 * cause a deadlock.  To get a deadlock, one would have to lock a
 * buffer followed by the dirhash while a second thread locked a
 * buffer while holding the dirhash lock.  The second order can happen
 * under a shared or exclusive vnode lock for the associated directory
 * in lookup().  The first order, however, can only happen under an
 * exclusive vnode lock (e.g. unlink(), rename(), etc.).  Thus, for
 * a thread to be doing a "bufwait" -> "dirhash" order, it has to hold
 * an exclusive vnode lock.  That exclusive vnode lock will prevent
 * any other threads from doing a "dirhash" -> "bufwait" order.

John Baldwin

More information about the freebsd-fs mailing list