FreeBSD 9 + ZFS + MPS

[Tim Gustafson]

I'm seeing some odd behavior on FreeBSD 9 with ZFS and an MPS controller.

For two or three weeks, this configuration was working like a champ.  It's a Bacula storage daemon server and was getting considerable usage - our backup set is in the multiple-terabytes-per-week of data being written to the disk.  However, during the last two weeks something has broken.  Starting on about July 15th or so we started to loose the data zpool which is connected via the mps driver.  The disks are an external array of 32 drives, etc 2TB.  When I did a "zpool status", I saw that some of the drives had dropped out of the array.  Rebooting the server brought them back until the next heavy write time.  On July 22nd I did a "make world" to see if there were any kernel updates that may have fixed the problem, but any updates that may have happened did not seem to help.

I'm including the last entry in my dmesg.yesterday file below.  Have I stumbled upon a bug?

lock order reversal:
 1st 0xfffffe0019cf6db8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:829
 2nd 0xfffffe0019cf69f8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2134
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
__lockmgr_args() at __lockmgr_args+0xd42
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x47
vget() at vget+0x7b
devfs_allocv() at devfs_allocv+0x13f
devfs_root() at devfs_root+0x4d
vfs_donmount() at vfs_donmount+0x988
nmount() at nmount+0x63
syscallenter() at syscallenter+0x1aa
syscall() at syscall+0x4c
Xfast_syscall() at Xfast_syscall+0xdd
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x800ab4dfc, rsp = 0x7fffffffccc8, rbp = 0x801009048 ---

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Tim Gustafson                                                tjg at soe.ucsc.edu
Baskin School of Engineering                                     831-459-5354
UC Santa Cruz                                         Baskin Engineering 317B
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-