ZFS and sh(1) panic: spin lock [lock addr] (smp rendezvous) held by [sh(1) proc tid] too long

[Brandon Gooch]

I've a few more details now (the original subject seems a little off now).

I added INVARIANTS and WITNESS to my kernel config.

I feel the first textdump wasn't as revealing as this most recent one.

>From the most recent textdump:

spin lock 0xffffffff80852940 (smp rendezvous) held by 0xffffff0005eeaae0 (tid 10
0170) too long
panic: spin lock held too long
cpuid = 2
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
_mtx_lock_spin_failed() at _mtx_lock_spin_failed+0x61
_mtx_lock_spin() at _mtx_lock_spin+0xb2
_mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x149
smp_rendezvous_cpus() at smp_rendezvous_cpus+0xd3
RTMpPokeCpu() at RTMpPokeCpu+0x35
g_aAdapters() at g_aAdapters+0x54b2
g_aAdapters() at 0xffffffff81e64795
g_aAdapters() at 0xffffffff81e64a7b
supdrvIOCtl() at supdrvIOCtl+0x17bd
VBoxDrvFreeBSDIOCtl() at VBoxDrvFreeBSDIOCtl+0x1f6
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1e4
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800c8c9fc, rsp = 0x7fffff46cbe8,
 rbp = 0x7fffff46cbf0 ---
KDB: enter: panic
exclusive sx IPRT Fast Mutex Semaphore (IPRT Fast Mutex Semaphore) r = 0 (0xffff
ff0007123a98) locked @ /usr/ports/emulators/virtualbox-ose-kmod/work/VirtualBox-
3.1.2_OSE/out/freebsd.amd64/release/bin/src/vboxdrv/r0drv/freebsd/semfastmutex-r
0drv-freebsd.c:103
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0007d8d648) locked @ /usr/src/
sys/kern/uipc_sockbuf.c:148
exclusive sleep mutex vm object (standard object) r = 0 (0xffffff0007ba3948) loc
ked @ /usr/src/sys/vm/vm_fault.c:297
shared sx user map (user map) r = 0 (0xffffff0007268200) locked @
/usr/src/sys/vm/vm_map.c:3532
exclusive sleep mutex pmap (pmap) r = 0 (0xffffffff80a4c3c0) locked @
/usr/src/sys/amd64/amd64/pmap.c:2463
exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0
(0xffffffff80a02940) locked @ /usr/src/sys/amd64/amd64/pmap.c:2462
exclusive sleep mutex system map (system map) r = 0
(0xffffff0002000180) locked @ /usr/src/sys/vm/vm_map.c:2772
exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff8082fec0) locked
@ /usr/src/sys/kern/kern_sysctl.c:1417
shared sx sysctl lock (sysctl lock) r = 0 (0xffffffff80830860) locked
@ /usr/src/sys/kern/kern_sysctl.c:1521

I can make the entire textdump available on request.

Also, I can provide any other detail necessary to help solve this,
including access to the box.

Thanks to whomever may be able to take a look at this!

-Brandon