"magic" crashes - mostly solved but

Wed Jun 27 21:55:33 UTC 2012

the reason was most probably of of date vbox and fuse kernel modules.

after making everything in sync system boots successfully with WITNESS, 
INVARIANT etc. options enabled.

STILL - mostly at booting i'm getting few messages.

first comes when executing /etc/rc.d/named (at mounting devfs IMHO):

Jun 27 18:32:23 foo kernel: lock order reversal:
Jun 27 18:32:23 foo kernel: 1st 0xffffff80f5859800 bufwait (bufwait) @/usr/src/sys/kern/vfs_bio.c:2636
Jun 27 18:32:24 foo kernel: 
Jun 27 18:32:24 foo kernel: 2nd 0xffffff0005c82200 dirhash (dirhash) @/usr/src/sys/ufs/ufs/ufs_dirhash.c:285
Jun 27 18:32:24 foo kernel: KDB: stack backtrace:
Jun 27 18:32:24 foo kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x27
Jun 27 18:32:24 foo kernel: em0: link state changed to UP
Jun 27 18:32:24 foo kernel: kdb_backtrace() at kdb_backtrace+0x3e
Jun 27 18:32:24 foo kernel: _witness_debugger() at _witness_debugger+0x24
Jun 27 18:32:24 foo kernel: witness_checkorder() at witness_checkorder+0xae7
Jun 27 18:32:24 foo kernel: _sx_xlock() at _sx_xlock+0xbf 
Jun 27 18:32:24 foo kernel: ufsdirhash_acquire() at ufsdirhash_acquire+0x4f
Jun 27 18:32:24 foo kernel: ufsdirhash_remove() at ufsdirhash_remove+0x1c
Jun 27 18:32:24 foo kernel: ufs_dirremove() at ufs_dirremove+0x12c
Jun 27 18:32:24 foo kernel: ufs_remove() at ufs_remove+0x8f
Jun 27 18:32:24 foo kernel: VOP_REMOVE_APV() at VOP_REMOVE_APV+0xf4
Jun 27 18:32:24 foo kernel: VOP_REMOVE() at VOP_REMOVE+0x45
Jun 27 18:32:24 foo kernel: kern_unlinkat() at kern_unlinkat+0x1ce
Jun 27 18:32:24 foo kernel: kern_unlink() at kern_unlink+0x28
Jun 27 18:32:24 foo kernel: unlink() at unlink+0x25
Jun 27 18:32:24 foo kernel: syscallenter() at syscallenter+0x2e3
Jun 27 18:32:24 foo kernel: amd64_syscall() at amd64_syscall+0x58
Jun 27 18:32:24 foo kernel: 
Jun 27 18:32:24 foo kernel: Xfast_syscall() at Xfast_syscall+0xfc
Jun 27 18:32:24 foo kernel: --- syscall (10, FreeBSD ELF64, unlink), rip = 0xeede070c, rsp = 0x7fffffffdb08, rbp = 0x7fffffffef58 ---
Jun 27 18:32:24 foo kernel: lock order reversal:
Jun 27 18:32:24 foo kernel: 1st 0xffffff00080a8270 ufs (ufs) @/usr/src/sys/kern/vfs_mount.c:1081
Jun 27 18:32:24 foo kernel: 2nd 0xffffff00085397f8 devfs (devfs) @/ /usr/src/sys/kern/vfs_subr.c:2169
Jun 27 18:32:24 foo kernel: KDB: stack backtrace:
Jun 27 18:32:24 foo kernel: db_trace_self_wrapper() atdb_trace_self_wrapper+0x27
Jun 27 18:32:24 foo kernel: kdb_backtrace() at kdb_backtrace+0x3e
Jun 27 18:32:24 foo kernel: _witness_debugger() at _witness_debugger+0x24
Jun 27 18:32:24 foo kernel: witness_checkorder() atwitness_checkorder+0xae7
Jun 27 18:32:24 foo kernel: __lockmgr_args() at __lockmgr_args+0x68d
Jun 27 18:32:24 foo kernel: _lockmgr_args() at _lockmgr_args+0x6f
Jun 27 18:32:24 foo kernel: vop_stdlock() at vop_stdlock+0x67
Jun 27 18:32:24 foo kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfd
Jun 27 18:32:24 foo kernel: VOP_LOCK1() at VOP_LOCK1+0x4b
Jun 27 18:32:24 foo kernel: _vn_lock() at _vn_lock+0x64
Jun 27 18:32:24 foo kernel: vget() at vget+0xe9
Jun 27 18:32:24 foo kernel: devfs_allocv() at devfs_allocv+0x125
Jun 27 18:32:24 foo kernel: devfs_root() at devfs_root+0x5a
Jun 27 18:32:24 foo kernel: vfs_domount() at vfs_domount+0xcdb
Jun 27 18:32:24 foo kernel: vfs_donmount() at vfs_donmount+0x78e
Jun 27 18:32:24 foo kernel: nmount() at nmount+0x7e
Jun 27 18:32:24 foo kernel: syscallenter() at syscallenter+0x2e3
Jun 27 18:32:24 foo kernel: amd64_syscall() at amd64_syscall+0x58
Jun 27 18:32:24 foo kernel: Xfast_syscall() at Xfast_syscall+0xfc
Jun 27 18:32:24 foo kernel: --- syscall (378, FreeBSD ELF64, nmount), rip= 0xeee6535c, rsp = 0x7fffffffdd18, rbp = 0xef206048 ---
Jun 27 18:32:24 foo named[1071]: starting BIND 9.6.-ESV-R7-P1 -t/var/named -u bind
Jun 27 18:32:24 foo kernel: Starting named.

few more when mounting or unmounting (i'm not sure) pendrive.

Jun 27 18:57:09 foo kernel: lock order reversal:
Jun 27 18:57:09 foo kernel: 1st 0xffffff011ec78098 ufs (ufs) @ /usr/src/sys/kern/vfs_lookup.c:504
Jun 27 18:57:09 foo kernel: 2nd 0xffffff80f5e1bb80 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_softdep.c:6193
Jun 27 18:57:09 foo kernel: 3rd 0xffffff011ead3d80 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2169
Jun 27 18:57:09 foo kernel: KDB: stack backtrace:
Jun 27 18:57:09 foo kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x27
Jun 27 18:57:09 foo kernel: kdb_backtrace() at kdb_backtrace+0x3e
Jun 27 18:57:09 foo kernel: _witness_debugger() at _witness_debugger+0x24
Jun 27 18:57:09 foo kernel: witness_checkorder() at witness_checkorder+0xae7
Jun 27 18:57:09 foo kernel: __lockmgr_args() at __lockmgr_args+0x68d
Jun 27 18:57:09 foo kernel: _lockmgr_args() at _lockmgr_args+0x6f
Jun 27 18:57:09 foo kernel: ffs_lock() at ffs_lock+0xaa
Jun 27 18:57:09 foo kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfd
Jun 27 18:57:09 foo kernel: VOP_LOCK1() at VOP_LOCK1+0x4b
Jun 27 18:57:09 foo kernel: _vn_lock() at _vn_lock+0x64
Jun 27 18:57:09 foo kernel: vget() at vget+0xe9
Jun 27 18:57:09 foo kernel: vfs_hash_get() at vfs_hash_get+0xe6
Jun 27 18:57:09 foo kernel: ffs_vgetf() at ffs_vgetf+0x4a
Jun 27 18:57:09 foo kernel: flush_pagedep_deps() at flush_pagedep_deps+0x12a
Jun 27 18:57:09 foo kernel: softdep_sync_metadata() at softdep_sync_metadata+0x474
Jun 27 18:57:09 foo kernel: ffs_syncvnode() at ffs_syncvnode+0x447
Jun 27 18:57:09 foo kernel: ffs_truncate() at ffs_truncate+0x786
Jun 27 18:57:09 foo kernel: ufs_direnter() at ufs_direnter+0xb1f
Jun 27 18:57:09 foo kernel: ufs_mkdir() at ufs_mkdir+0x8d8
Jun 27 18:57:09 foo kernel: VOP_MKDIR_APV() at VOP_MKDIR_APV+0xf4
Jun 27 18:57:09 foo kernel: VOP_MKDIR() at VOP_MKDIR+0x51
Jun 27 18:57:09 foo kernel: kern_mkdirat() at kern_mkdirat+0x232
Jun 27 18:57:09 foo kernel: kern_mkdir() at kern_mkdir+0x31
Jun 27 18:57:09 foo kernel: mkdir() at mkdir+0x20
Jun 27 18:57:09 foo kernel: syscallenter() at syscallenter+0x2e3
Jun 27 18:57:09 foo kernel: amd64_syscall() at amd64_syscall+0x58
Jun 27 18:57:09 foo kernel: Xfast_syscall() at Xfast_syscall+0xfc
Jun 27 18:57:09 foo kernel: --- syscall (136, FreeBSD ELF64, mkdir), rip = 0xeede1dfc, rsp = 0x7fffffffe998, rbp = 0x7fffffffed16 ---

any idea what is it?