ZFS v28 and the infamous deadlock

Fri Jan 14 11:47:08 UTC 2011

Hello,

Long ago I reported a deadlock condition. I reached it by sending incremental snapshots to a second machine while doing heavy read access on the target dataset.  (I include my original posting below for reference).

I've tried the same with the new ZFS v28 and it's been able to run for some hours without deadlock (hooray!) but it has crashed with a panic. However, I was running both FreeBSD systems under VMWare Fusion on my laptop, and the memory was scarce for them, so it could be related to a memory shortage.

I'm going to set up a couple of servers and try it anyway. At least seems there's an important change. 

Pawel, did you try to reproduce the infamous deadlock with v28?

Borja.

(original message follows)

Trying. I started my typical test:

Machine 1 doing a make buildworld on a dataset with src and obj on it.

Machine 1 replicating incremental snapshots of the dataset to machine 2.

Machine 2 running some "tar cf - . | ( cd /pool/anotherdataset && tar xf - )" from the dataset being replicated, ie, doing read operations on the target dataset.

This time, with all those debug options, there was no deadlock, but almost an instant trap entering DDB. Unfortunately, I tried to capture the output of "alltrace", etc using the capture option. But couldn't come back to the system to read it.

Any ideas? I'm using VMWare Fusion to run FreeBSD for these tests and seems I'm out of luck, I don't see any console output mechanism.

When rebooting I was greeted by some LORs
lock order reversal:
1st 0xffffff000286c2e8 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:549
2nd 0xffffff000286b0d8 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:1173
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_block_freed() at dnode_block_freed+0x8e
dbuf_read() at dbuf_read+0x155
dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a
dmu_read() at dmu_read+0x80
load_nvlist() at load_nvlist+0x85
spa_load() at spa_load+0x49a
spa_open_common() at spa_open_common+0x12d
spa_get_stats() at spa_get_stats+0x42
zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe7d1c, rsp = 0x7fffffffd808, rbp = 0x801224140 ---
lock order reversal:
1st 0xffffff0002864e70 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:381
2nd 0xffffff00026e5140 osi->os_lock (osi->os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:323
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_destroy() at dnode_destroy+0xa6
dnode_buf_pageout() at dnode_buf_pageout+0xb2
dbuf_evict_user() at dbuf_evict_user+0x55
dbuf_clear() at dbuf_clear+0x5e
dnode_evict_dbufs() at dnode_evict_dbufs+0x98
dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c
dmu_objset_evict() at dmu_objset_evict+0xbf
dsl_pool_close() at dsl_pool_close+0x52
spa_unload() at spa_unload+0xb2
spa_load() at spa_load+0x4da
spa_open_common() at spa_open_common+0x12d
spa_get_stats() at spa_get_stats+0x42
zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe7d1c, rsp = 0x7fffffffd808, rbp = 0x801224140 ---
lock order reversal:
1st 0xffffff000286c058 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1116
2nd 0xffffff0002591a38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1120
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_dirty() at dbuf_dirty+0x892
dnode_setdirty() at dnode_setdirty+0x1a9
dbuf_dirty() at dbuf_dirty+0xa53
bplist_vacate() at bplist_vacate+0x4d
spa_sync() at spa_sync+0x297
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff0002905638 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1905
2nd 0xffffff000250c2f0 spa->spa_sync_bplist.bpl_lock (spa->spa_sync_bplist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:235
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47
zio_free() at zio_free+0x105
arc_free() at arc_free+0x11c
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483
dbuf_write() at dbuf_write+0x24c
dbuf_sync_list() at dbuf_sync_list+0x3eb
dbuf_sync_list() at dbuf_sync_list+0x17f
dnode_sync() at dnode_sync+0xc12
dmu_objset_sync() at dmu_objset_sync+0x134
dsl_pool_sync() at dsl_pool_sync+0x200
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff000293bcc8 zfs (zfs) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/gfs.c:437
2nd 0xffffff0002916310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:866
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x23c
zfs_root() at zfs_root+0x50
zfsctl_create() at zfsctl_create+0x82
zfs_mount() at zfs_mount+0x7ef
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x800f48f1c, rsp = 0x7fffffffced8, rbp = 0x7fffffffcef8 ---
lock order reversal:
1st 0xffffff00025e4078 zp->z_name_lock (zp->z_name_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:212
2nd 0xffffff0002916330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:866
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zget() at zfs_zget+0x23c
zfs_dirent_lock() at zfs_dirent_lock+0x4a0
zfs_dirlook() at zfs_dirlook+0x90
zfs_lookup() at zfs_lookup+0x256
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x3d3
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd8acc, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd3d0 ---
lock order reversal:
1st 0xffffff0002916210 zfsvfs->z_teardown_inactive_lock (zfsvfs->z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3724
2nd 0xffffff0002916330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1027
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_zinactive() at zfs_zinactive+0x95
zfs_inactive() at zfs_inactive+0x7e
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5
vinactive() at vinactive+0x90
vputx() at vputx+0x2fc
kern_statat_vnhook() at kern_statat_vnhook+0xfa
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd8acc, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd3d0 ---
lock order reversal:
1st 0xffffff0002861958 buf->b_lock (buf->b_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2509
2nd 0xffffff0002956430 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:421
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_read_done() at dbuf_read_done+0x3b
arc_read_done() at arc_read_done+0x1d2
zio_done() at zio_done+0x308
zio_execute() at zio_execute+0xb1
arc_read_nolock() at arc_read_nolock+0x3d0
arc_read() at arc_read+0xaf
dbuf_read() at dbuf_read+0x62b
dmu_buf_hold() at dmu_buf_hold+0xcc
zap_lockdir() at zap_lockdir+0x6e
zap_cursor_retrieve() at zap_cursor_retrieve+0x1bc
zfs_unlinked_drain() at zfs_unlinked_drain+0xd8
zfsvfs_setup() at zfsvfs_setup+0xfa
zfs_mount() at zfs_mount+0x7df
vfs_donmount() at vfs_donmount+0xcde
nmount() at nmount+0x63
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x800f48f1c, rsp = 0x7fffffffced8, rbp = 0x7fffffffcef8 ---
Expensive timeout(9) function: 0xffffffff80329580(0xffffff8000284000) 0.012686807 s
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to deny, logging disabled
lock order reversal:
1st 0xffffff800a2f5d08 bufwait (bufwait) @ /pool/newsrc/src/sys/kern/vfs_bio.c:2559
2nd 0xffffff0002c69800 dirhash (dirhash) @ /pool/newsrc/src/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
ufsdirhash_acquire() at ufsdirhash_acquire+0x44
ufsdirhash_add() at ufsdirhash_add+0x19
ufs_direnter() at ufs_direnter+0x88b
ufs_mkdir() at ufs_mkdir+0x623
VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9
kern_mkdirat() at kern_mkdirat+0x264
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (136, FreeBSD ELF64, mkdir), rip = 0x80072bb0c, rsp = 0x7fffffffec88, rbp = 0x7fffffffef66 ---
lock order reversal:
1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:189
2nd 0xffffff000286ab88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:130
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_slock() at _sx_slock+0x55
dnode_verify() at dnode_verify+0x70
dnode_hold_impl() at dnode_hold_impl+0x73
dmu_bonus_hold() at dmu_bonus_hold+0x31
bplist_hold() at bplist_hold+0x48
bplist_enqueue() at bplist_enqueue+0x4c
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119
dmu_objset_sync() at dmu_objset_sync+0x1fe
dsl_pool_sync() at dsl_pool_sync+0x88
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:189
2nd 0xffffff0002f6b0d8 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:606
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_hold_impl() at dnode_hold_impl+0x184
dmu_bonus_hold() at dmu_bonus_hold+0x31
bplist_hold() at bplist_hold+0x48
bplist_enqueue() at bplist_enqueue+0x4c
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119
dmu_objset_sync() at dmu_objset_sync+0x1fe
dsl_pool_sync() at dsl_pool_sync+0x88
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:189
2nd 0xffffff00028842e8 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1724
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_rele() at dbuf_rele+0x2d
dnode_hold_impl() at dnode_hold_impl+0x20f
dmu_bonus_hold() at dmu_bonus_hold+0x31
bplist_hold() at bplist_hold+0x48
bplist_enqueue() at bplist_enqueue+0x4c
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119
dmu_objset_sync() at dmu_objset_sync+0x1fe
dsl_pool_sync() at dsl_pool_sync+0x88
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:189
2nd 0xffffff00027d5d40 osi->os_lock (osi->os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:687
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dnode_setdirty() at dnode_setdirty+0xbc
dbuf_dirty() at dbuf_dirty+0x516
bplist_enqueue() at bplist_enqueue+0xbd
dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119
dmu_objset_sync() at dmu_objset_sync+0x1fe
dsl_pool_sync() at dsl_pool_sync+0x88
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff0002c84938 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1905
2nd 0xffffff0002f6b7b0 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:543
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_slock() at _sx_slock+0x55
dbuf_read() at dbuf_read+0x2ad
dbuf_will_dirty() at dbuf_will_dirty+0x53
dsl_dataset_block_kill() at dsl_dataset_block_kill+0xe9
dbuf_write() at dbuf_write+0x24c
dbuf_sync_list() at dbuf_sync_list+0x159
dbuf_sync_list() at dbuf_sync_list+0x17f
dnode_sync() at dnode_sync+0xc12
dmu_objset_sync() at dmu_objset_sync+0x134
dsl_pool_sync() at dsl_pool_sync+0x88
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff000291a210 zfsvfs->z_teardown_inactive_lock (zfsvfs->z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:917
2nd 0xffffff00027450f8 ds->ds_rwlock (ds->ds_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:2864
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dsl_dataset_clone_swap() at dsl_dataset_clone_swap+0x5a
dmu_recv_end() at dmu_recv_end+0x94
zfs_ioc_recv() at zfs_ioc_recv+0x29d
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe7d1c, rsp = 0x7fffffff8e98, rbp = 0x7fffffff9bd0 ---
lock order reversal:
1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:94
2nd 0xffffff0002f6b330 dn->dn_dbufs_mtx (dn->dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1518
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_destroy() at dbuf_destroy+0x58
bplist_close() at bplist_close+0x37
dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0x506
dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173
dsl_pool_sync() at dsl_pool_sync+0x122
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:94
2nd 0xffffffff81152650 h->hash_mutexes[i] (h->hash_mutexes[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:191
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
dbuf_destroy() at dbuf_destroy+0x111
bplist_close() at bplist_close+0x37
dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0x506
dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173
dsl_pool_sync() at dsl_pool_sync+0x122
spa_sync() at spa_sync+0x35e
txg_sync_thread() at txg_sync_thread+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 ---
lock order reversal:
1st 0xffffff000291a250 zfsvfs->z_znodes_lock (zfsvfs->z_znodes_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1317
2nd 0xffffff000291a310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:966
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_sx_xlock() at _sx_xlock+0x55
zfs_rezget() at zfs_rezget+0x4a
zfs_resume_fs() at zfs_resume_fs+0x158
zfs_ioc_recv() at zfs_ioc_recv+0x2b4
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe7d1c, rsp = 0x7fffffff8e98, rbp = 0x7fffffff9bd0 ---
# 

Borja.