FreeBSD 9 + ZFS + MPS

[Eric McCorkle]

On 7/25/11 11:32 AM, Tim Gustafson wrote:
> I'm seeing some odd behavior on FreeBSD 9 with ZFS and an MPS controller.
>
> For two or three weeks, this configuration was working like a champ.  It's a Bacula storage daemon server and was getting considerable usage - our backup set is in the multiple-terabytes-per-week of data being written to the disk.  However, during the last two weeks something has broken.  Starting on about July 15th or so we started to loose the data zpool which is connected via the mps driver.  The disks are an external array of 32 drives, etc 2TB.  When I did a "zpool status", I saw that some of the drives had dropped out of the array.  Rebooting the server brought them back until the next heavy write time.  On July 22nd I did a "make world" to see if there were any kernel updates that may have fixed the problem, but any updates that may have happened did not seem to help.
>
> I'm including the last entry in my dmesg.yesterday file below.  Have I stumbled upon a bug?
>
> lock order reversal:
>   1st 0xfffffe0019cf6db8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:829
>   2nd 0xfffffe0019cf69f8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2134
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> kdb_backtrace() at kdb_backtrace+0x37
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x807
> __lockmgr_args() at __lockmgr_args+0xd42
> vop_stdlock() at vop_stdlock+0x39
> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
> _vn_lock() at _vn_lock+0x47
> vget() at vget+0x7b
> devfs_allocv() at devfs_allocv+0x13f
> devfs_root() at devfs_root+0x4d
> vfs_donmount() at vfs_donmount+0x988
> nmount() at nmount+0x63
> syscallenter() at syscallenter+0x1aa
> syscall() at syscall+0x4c
> Xfast_syscall() at Xfast_syscall+0xdd
> --- syscall (378, FreeBSD ELF64, nmount), rip = 0x800ab4dfc, rsp = 0x7fffffffccc8, rbp = 0x801009048 ---
>

I've seen similar messages on a ZFS-based system, when rebooting, and 
when unmounting a USB memory stick (curiously, with an msdosfs 
filesystem, though my hard drive contains ZFS)