UFS Witness LoR + 5 other LoRs

[Kris Kennaway]

Martin wrote:
> Am Mon, 26 Jan 2009 14:01:03 -0800
> schrieb Sean Bruno <sean.bruno at dsl-only.net>:
> 
>> Anyone else seeing this LoR today?
>>
>> lock order reversal:
>>  1st 0xc456b044 user map (user map)
>> @ /root/bsd/head/sys/vm/vm_map.c:3198
>>  2nd 0xc4911ad0 ufs (ufs) @ /root/bsd/head/sys/kern/vfs_subr.c:2071
>> KDB: stack backtrace:
>> db_trace_self_wrapper(c0bf0913,c417990c,c087a4b5,c0c1458c,0,...) at
>> db_trace_self_wrapper+0x26
>> kdb_backtrace(c0c1458c,0,c4521728,c4526730,c4179968,...) at
>> kdb_backtrace+0x29
>> _witness_debugger(c0bf3680,c4911ad0,c0be6d88,c4526730,c0bfa429,...) at
>> _witness_debugger+0x25
>> witness_checkorder(c4911ad0,1,c0bfa429,817,0,...) at
>> witness_checkorder +0x839
>> __lockmgr_args(c4911ad0,200501,c4911aec,0,0,...) at
>> __lockmgr_args+0x237
>> ffs_lock(c4179a78,c087a25b,c0c1697a,200501,c4911a78,...) at ffs_lock
>> +0x8a VOP_LOCK1_APV(c0cf7ca0,c4179a78,c4567e24,c0d0bdc0,c4911a78,...)
>> at VOP_LOCK1_APV+0xb5
>> _vn_lock(c4911a78,200501,c0bfa429,817,4,...) at _vn_lock+0x5e
>> vget(c4911a78,200501,c4567d80,4b4,0,...) at vget+0xc9
>> vnode_pager_lock(c187c1f0,0,c0c13f07,127,c4179c18,...) at
>> vnode_pager_lock+0x1e0
>> vm_fault(c456b000,80db000,2,8,80db620,...) at vm_fault+0x1df
>> trap_pfault(5,0,c0c241d1,2e7,c4565d34,...) at trap_pfault+0x118
>> trap(c4179d38) at trap+0x289
>> calltrap() at calltrap+0x6
>> --- trap 0xc, eip = 0x80480e5, esp = 0xbfbfeef0, ebp = 0xbfbfef10 ---
> 
> Hi,
> 
> I have similar LORs. One of them might be affecting Gnome2. I cannot
> even save anything once the LOR happened. nautilus is also hanging
> around. I cannot even kill it. This occurs since I migrated
> my /usr/home to ZFS (see fourth, fifth LoR).
> 
> Here one like yours:
> 
> lock order reversal:
>  1st 0xffffff0002402070 user map (user map)
> @ /usr/src/sys/vm/vm_map.c:3198 2nd 0xffffff0002b8a7f8 ufs (ufs)
> @ /usr/src/sys/kern/vfs_subr.c:2071
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x81e
> __lockmgr_args() at __lockmgr_args+0xca8
> ffs_lock() at ffs_lock+0x8c
> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
> _vn_lock() at _vn_lock+0x47
> vget() at vget+0x8b
> vnode_pager_lock() at vnode_pager_lock+0x1d0
> vm_fault() at vm_fault+0x1e2
> trap_pfault() at trap_pfault+0x128
> trap() at trap+0x51c
> calltrap() at calltrap+0x8
> --- trap 0xc, rip = 0x40014f, rsp = 0x7fffffffee70, rbp =
> 0x7fffffffee90 ---
> 
> 
> Here a second one:
> 
> lock order reversal:
>  1st 0xfffffffe803eb610 bufwait (bufwait)
> @ /usr/src/sys/kern/vfs_bio.c:2443 2nd 0xffffff00080d2000 dirhash
> (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:275
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x81e
> _sx_xlock() at _sx_xlock+0x55
> ufsdirhash_acquire() at ufsdirhash_acquire+0x33
> ufsdirhash_add() at ufsdirhash_add+0x19
> ufs_direnter() at ufs_direnter+0x88b
> ufs_mkdir() at ufs_mkdir+0x633
> VOP_MKDIR_APV() at VOP_MKDIR_APV+0x93
> kern_mkdirat() at kern_mkdirat+0x2b1
> syscall() at syscall+0x1bf
> Xfast_syscall() at Xfast_syscall+0xab
> --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x800719e9c, rsp =
> 0x7fffffffed08, rbp = 0x7fffffffef76 ---
> 
> 
> Here the third:
> 
> lock order reversal:
>  1st 0xffffff0008648248 filedesc structure (filedesc structure)
> @ /usr/src/sys/kern/kern_descrip.c:1076 2nd 0xffffff0019110ba8 ufs
> (ufs) @ /usr/src/sys/kern/vfs_subr.c:4057
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x81e
> __lockmgr_args() at __lockmgr_args+0xc2a
> ffs_lock() at ffs_lock+0x8c
> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
> _vn_lock() at _vn_lock+0x47
> knlist_remove_kq() at knlist_remove_kq+0x73
> knote_fdclose() at knote_fdclose+0x177
> kern_close() at kern_close+0xe6
> syscall() at syscall+0x1bf
> Xfast_syscall() at Xfast_syscall+0xab
> --- syscall (6, FreeBSD ELF64, close), rip = 0x800e35e8c, rsp =
> 0x7fffffffe678, rbp = 0x801063a00 ---
> 
> 
> 
> And thes two are really nasty (ZFS):
> 
> lock order reversal:
>  1st 0xffffff00195ed620 zfs (zfs) @ /usr/src/sys/kern/vfs_vnops.c:531
>  2nd 0xffffff00080ab360 user map (user map)
> @ /usr/src/sys/vm/vm_map.c:3198 KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x81e
> _sx_slock() at _sx_slock+0x55
> vm_map_lookup() at vm_map_lookup+0x47
> vm_fault() at vm_fault+0xfe
> trap_pfault() at trap_pfault+0x128
> trap() at trap+0x347
> calltrap() at calltrap+0x8
> --- trap 0xc, rip = 0xffffffff8081ed4b, rsp = 0xfffffffe97261850, rbp =
> 0xfffffffe972618d0 --- copyout() at copyout+0x3b
> dmu_read_uio() at dmu_read_uio+0x98
> zfs_freebsd_read() at zfs_freebsd_read+0x56d
> vn_read() at vn_read+0x267
> dofileread() at dofileread+0xa1
> kern_readv() at kern_readv+0x60
> read() at read+0x54
> syscall() at syscall+0x1bf
> Xfast_syscall() at Xfast_syscall+0xab
> --- syscall (3, FreeBSD ELF64, read), rip = 0x80647becc, rsp =
> 0x7ffffffed8c8, rbp = 0x806682f80 ---
> 
> lock order reversal:
>  1st 0xffffff0019bc2270 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2071
>  2nd 0xffffff0019a77448 filedesc structure (filedesc structure)
> @ /usr/src/sys/kern/vfs_syscalls.c:1720
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x81e
> _sx_slock() at _sx_slock+0x55
> kern_symlinkat() at kern_symlinkat+0x1f8
> syscall() at syscall+0x1bf
> Xfast_syscall() at Xfast_syscall+0xab
> --- syscall (57, FreeBSD ELF64, symlink), rip = 0x8049cb13c, rsp =
> 0x7fffffffd6f8, rbp = 0x7fffffffdd10 ---
> 
> And one with NFS:
> 
> lock order reversal:
>  1st 0xffffff00080a6940 user map (user map)
> @ /usr/src/sys/vm/vm_map.c:3198 2nd 0xffffff00566a9ba8 nfs (nfs)
> @ /usr/src/sys/kern/vfs_subr.c:2071
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x81e
> __lockmgr_args() at __lockmgr_args+0xca8
> vop_stdlock() at vop_stdlock+0x39
> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
> _vn_lock() at _vn_lock+0x47
> vget() at vget+0x8b
> vnode_pager_lock() at vnode_pager_lock+0x1d0
> vm_fault() at vm_fault+0x1e2
> trap_pfault() at trap_pfault+0x128
> trap() at trap+0x347
> calltrap() at calltrap+0x8
> --- trap 0xc, rip = 0xffffffff8081edcd, rsp = 0xfffffffe972b67d0, rbp =
> 0xfffffffe972b6850 --- copyin() at copyin+0x3d
> ffs_write() at ffs_write+0x2f8
> VOP_WRITE_APV() at VOP_WRITE_APV+0xfe
> vn_write() at vn_write+0x23f
> dofilewrite() at dofilewrite+0x85
> kern_writev() at kern_writev+0x60
> write() at write+0x54
> syscall() at syscall+0x1bf
> Xfast_syscall() at Xfast_syscall+0xab
> --- syscall (4, FreeBSD ELF64, write), rip = 0x80072beac, rsp =
> 0x7fffffffe5d8, rbp = 0x800537000 ---
> 
> 
> Can you beat my 6 LoRs? ;)

Several of these are widely reported, and harmless (not sure about the 
filedesc ones though).  Which one do you think is causing your problem?

Kris