8.0-RC2: ZFS deadlock with zfs receive
Borja Marcos
borjam at sarenet.es
Mon Nov 2 11:02:04 UTC 2009
(Resending to the list, I replied only to Pawel)
On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote:
> On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote:
>> Borja Marcos wrote:
>>> I've been sending some alltraces to pjd about this easy to reproduce
>>> problem. I am using zfs send/zfs receive to replicate a dataset
>>> from one
>>> server to another. At 1 minute intervals, an incremental snapshot is
>>> sent to update the dataset copy. If there is reading activity on the
>>> dataset copy, a deadlock can happen rendering ZFS and all the FS
>>> subsystem unusable. I've tried with 8.0RC2 and it still happens.
>
> I was able to reproduce it, but I don't have fix yet.
>
>> FWIW, another (or the same) zfs recv deadlock I've been trying to
>> get to
>> the bottom of:
>>
>> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/
>> 006999.html
>
> Could you guys recompile your kernel after uncommenting line:
>
> #CFLAGS+=-DDEBUG=1
>
> in sys/modules/zfs/Makefile?
>
> You should see panic on assertion instead of deadlock, I believe.
No panic, it just deadlocked.
Last trace of LORs
Nov 2 13:45:44 kernel: lock order reversal:
Nov 2 13:45:44 kernel: 1st 0xffffff00025d4c38 ds-
>ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
bplist.c:189
Nov 2 13:45:44 kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi-
>os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dnode.c:705
Nov 2 13:45:44 kernel: KDB: stack backtrace:
Nov 2 13:45:44 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:45:44 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:45:44 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:45:44 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:45:44 kernel: dnode_setdirty() at dnode_setdirty+0xbc
Nov 2 13:45:44 kernel: dbuf_dirty() at dbuf_dirty+0x516
Nov 2 13:45:44 kernel: bplist_enqueue() at bplist_enqueue+0xbd
Nov 2 13:45:44 kernel: dsl_dataset_block_kill() at
dsl_dataset_block_kill+0x119
Nov 2 13:45:44 kernel: dmu_objset_sync() at dmu_objset_sync+0x1fe
Nov 2 13:45:44 kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov 2 13:45:44 kernel: spa_sync() at spa_sync+0x35e
Nov 2 13:45:44 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:45:44 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:45:44 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:45:44 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:45:44 kernel: lock order reversal:
Nov 2 13:45:44 kernel: 1st 0xffffff0002eb9d38 dr->dt.di.dr_mtx (dr-
>dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov 2 13:45:44 kernel: 2nd 0xffffff0002ed3000 dn->dn_struct_rwlock
(dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dbuf.c:543
Nov 2 13:45:44 kernel: KDB: stack backtrace:
Nov 2 13:45:44 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:45:44 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:45:44 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:45:44 kernel: _sx_slock() at _sx_slock+0x55
Nov 2 13:45:44 kernel: dbuf_read() at dbuf_read+0x2ad
Nov 2 13:45:44 kernel: dbuf_will_dirty() at dbuf_will_dirty+0x53
Nov 2 13:45:44 kernel: dsl_dataset_block_kill() at
dsl_dataset_block_kill+0xe9
Nov 2 13:45:44 kernel: dbuf_write() at dbuf_write+0x24c
Nov 2 13:45:44 kernel: dbuf_sync_list() at dbuf_sync_list+0x159
Nov 2 13:45:44 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov 2 13:45:44 kernel: dnode_sync() at dnode_sync+0xc12
Nov 2 13:45:44 kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov 2 13:45:44 kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov 2 13:45:44 kernel: spa_sync() at spa_sync+0x35e
Nov 2 13:45:44 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:45:44 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:45:44 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:45:44 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:46:03 kernel: lock order reversal:
Nov 2 13:46:03 kernel: 1st 0xffffff00025d4c38 ds-
>ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
bplist.c:152
Nov 2 13:46:03 kernel: 2nd 0xffffff0002f01ae0 dn->dn_dbufs_mtx (dn-
>dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1518
Nov 2 13:46:03 kernel: KDB: stack backtrace:
Nov 2 13:46:03 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:46:03 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:46:03 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:46:03 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:46:03 kernel: dbuf_destroy() at dbuf_destroy+0x58
Nov 2 13:46:03 kernel: bplist_cache() at bplist_cache+0x2e
Nov 2 13:46:03 kernel: bplist_iterate() at bplist_iterate+0xb3
Nov 2 13:46:03 kernel: bplist_space_birthrange() at
bplist_space_birthrange+0x60
Nov 2 13:46:03 kernel: dsl_dataset_clone_swap_sync() at
dsl_dataset_clone_swap_sync+0xee
Nov 2 13:46:03 kernel: dsl_sync_task_group_sync() at
dsl_sync_task_group_sync+0x173
Nov 2 13:46:03 kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov 2 13:46:03 kernel: spa_sync() at spa_sync+0x35e
Nov 2 13:46:03 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:46:03 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:46:03 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:46:03 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:46:03 kernel: lock order reversal:
Nov 2 13:46:03 kernel: 1st 0xffffff00025d4c38 ds-
>ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/
bplist.c:152
Nov 2 13:46:03 kernel: 2nd 0xffffffff81121d50 h->hash_mutexes[i] (h-
>hash_mutexes[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dbuf.c:191
Nov 2 13:46:03 kernel: KDB: stack backtrace:
Nov 2 13:46:03 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:46:03 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:46:03 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:46:03 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:46:03 kernel: dbuf_destroy() at dbuf_destroy+0x111
Nov 2 13:46:03 kernel: bplist_cache() at bplist_cache+0x2e
Nov 2 13:46:03 kernel: bplist_iterate() at bplist_iterate+0xb3
Nov 2 13:46:03 kernel: bplist_space_birthrange() at
bplist_space_birthrange+0x60
Nov 2 13:46:03 kernel: dsl_dataset_clone_swap_sync() at
dsl_dataset_clone_swap_sync+0xee
Nov 2 13:46:03 kernel: dsl_sync_task_group_sync() at
dsl_sync_task_group_sync+0x173
Nov 2 13:46:03 kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov 2 13:46:03 kernel: spa_sync() at spa_sync+0x35e
Nov 2 13:46:03 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:46:03 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:46:03 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:46:03 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:46:35 kernel: lock order reversal:
Nov 2 13:46:35 kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock
(dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171
Nov 2 13:46:35 kernel: 2nd 0xffffff0002784938 ds->ds_opening_lock
(ds->ds_opening_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:483
Nov 2 13:46:35 kernel: KDB: stack backtrace:
Nov 2 13:46:35 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:46:35 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:46:35 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:46:35 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:46:35 kernel: dmu_objset_create_impl() at
dmu_objset_create_impl+0x50
Nov 2 13:46:35 kernel: dmu_objset_create_sync() at
dmu_objset_create_sync+0xfc
Nov 2 13:46:35 kernel: dsl_sync_task_group_sync() at
dsl_sync_task_group_sync+0x173
Nov 2 13:46:35 kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov 2 13:46:35 kernel: spa_sync() at spa_sync+0x35e
Nov 2 13:46:35 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:46:35 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:46:35 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:46:35 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:46:35 kernel: lock order reversal:
Nov 2 13:46:35 kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock
(dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171
Nov 2 13:46:35 kernel: 2nd 0xffffff0002fb7578 zfs (zfs) @ /pool/
newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/
fs/zfs/zfs_znode.c:152
Nov 2 13:46:35 kernel: KDB: stack backtrace:
Nov 2 13:46:35 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:46:35 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:46:35 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:46:35 kernel: __lockmgr_args() at __lockmgr_args+0xd03
Nov 2 13:46:35 kernel: vop_stdlock() at vop_stdlock+0x39
Nov 2 13:46:35 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
Nov 2 13:46:35 kernel: _vn_lock() at _vn_lock+0x57
Nov 2 13:46:35 kernel: zfs_znode_cache_constructor() at
zfs_znode_cache_constructor+0x65
Nov 2 13:46:35 kernel: zfs_create_fs() at zfs_create_fs+0x2ab
Nov 2 13:46:35 kernel: dmu_objset_create_sync() at
dmu_objset_create_sync+0x116
Nov 2 13:46:35 kernel: dsl_sync_task_group_sync() at
dsl_sync_task_group_sync+0x173
Nov 2 13:46:35 kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov 2 13:46:35 kernel: spa_sync() at spa_sync+0x35e
Nov 2 13:46:35 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:46:35 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:46:35 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:46:35 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:46:36 kernel: lock order reversal:
Nov 2 13:46:36 kernel: 1st 0xffffff001a0a7430 db->db_mtx (db-
>db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1724
Nov 2 13:46:36 kernel: 2nd 0xffffff001a4c0708 dn->dn_dbufs_mtx (dn-
>dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dnode_sync.c:373
Nov 2 13:46:36 kernel: KDB: stack backtrace:
Nov 2 13:46:36 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:46:36 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:46:36 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:46:36 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:46:36 kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x57
Nov 2 13:46:36 kernel: dmu_objset_evict_dbufs() at
dmu_objset_evict_dbufs+0xd4
Nov 2 13:46:36 kernel: dmu_objset_evict() at dmu_objset_evict+0xbf
Nov 2 13:46:36 kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54
Nov 2 13:46:36 kernel: dbuf_evict_user() at dbuf_evict_user+0x55
Nov 2 13:46:36 kernel: dbuf_rele() at dbuf_rele+0x173
Nov 2 13:46:36 kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c
Nov 2 13:46:36 kernel: spa_sync() at spa_sync+0x618
Nov 2 13:46:36 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:46:36 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:46:36 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:46:36 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:46:36 kernel: lock order reversal:
Nov 2 13:46:36 kernel: 1st 0xffffff001a0a7430 db->db_mtx (db-
>db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1724
Nov 2 13:46:36 kernel: 2nd 0xffffff001a4c03d8 dn->dn_struct_rwlock
(dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:409
Nov 2 13:46:36 kernel: KDB: stack backtrace:
Nov 2 13:46:36 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:46:36 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:46:36 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:46:36 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:46:36 kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x1b8
Nov 2 13:46:36 kernel: dmu_objset_evict_dbufs() at
dmu_objset_evict_dbufs+0xd4
Nov 2 13:46:36 kernel: dmu_objset_evict() at dmu_objset_evict+0xbf
Nov 2 13:46:36 kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54
Nov 2 13:46:36 kernel: dbuf_evict_user() at dbuf_evict_user+0x55
Nov 2 13:46:36 kernel: dbuf_rele() at dbuf_rele+0x173
Nov 2 13:46:36 kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c
Nov 2 13:46:36 kernel: spa_sync() at spa_sync+0x618
Nov 2 13:46:36 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:46:36 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:46:36 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:46:36 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:46:57 kernel: lock order reversal:
Nov 2 13:46:57 kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr-
>dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov 2 13:46:57 kernel: 2nd 0xffffff0002fdb4b0 dn->dn_mtx (dn-
>dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1066
Nov 2 13:46:57 kernel: KDB: stack backtrace:
Nov 2 13:46:57 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:46:57 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:46:57 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:46:57 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:46:57 kernel: dbuf_dirty() at dbuf_dirty+0xa03
Nov 2 13:46:57 kernel: dsl_dataset_block_kill() at
dsl_dataset_block_kill+0x38b
Nov 2 13:46:57 kernel: dbuf_write() at dbuf_write+0x24c
Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb
Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov 2 13:46:57 last message repeated 5 times
Nov 2 13:46:57 kernel: dnode_sync() at dnode_sync+0xc12
Nov 2 13:46:57 kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov 2 13:46:57 kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov 2 13:46:57 kernel: spa_sync() at spa_sync+0x35e
Nov 2 13:46:57 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:46:57 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:46:57 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:46:57 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:46:57 kernel: lock order reversal:
Nov 2 13:46:57 kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr-
>dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov 2 13:46:57 kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi-
>os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/dnode.c:705
Nov 2 13:46:57 kernel: KDB: stack backtrace:
Nov 2 13:46:57 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:46:57 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:46:57 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:46:57 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:46:57 kernel: dnode_setdirty() at dnode_setdirty+0xbc
Nov 2 13:46:57 kernel: dbuf_dirty() at dbuf_dirty+0xa53
Nov 2 13:46:57 kernel: dsl_dataset_block_kill() at
dsl_dataset_block_kill+0x38b
Nov 2 13:46:57 kernel: dbuf_write() at dbuf_write+0x24c
Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb
Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov 2 13:46:57 last message repeated 5 times
Nov 2 13:46:57 kernel: dnode_sync() at dnode_sync+0xc12
Nov 2 13:46:57 kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov 2 13:46:57 kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov 2 13:46:57 kernel: spa_sync() at spa_sync+0x35e
Nov 2 13:46:57 kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov 2 13:46:57 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:46:57 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:46:57 kernel: --- trap 0, rip = 0, rsp =
0xffffff80001ffd30, rbp = 0 ---
Nov 2 13:47:28 kernel: lock order reversal:
Nov 2 13:47:28 kernel: 1st 0xffffff0002fb6a58 syncer (syncer) @ /
pool/newsrc/src/sys/kern/vfs_subr.c:1693
Nov 2 13:47:28 kernel: 2nd 0xffffff001a868350 zfsvfs->z_hold_mtx[i]
(zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863
Nov 2 13:47:28 kernel: KDB: stack backtrace:
Nov 2 13:47:28 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:47:28 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:47:28 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:47:28 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:47:28 kernel: zfs_zget() at zfs_zget+0x23a
Nov 2 13:47:28 kernel: zfs_get_data() at zfs_get_data+0x5e
Nov 2 13:47:28 kernel: zil_commit() at zil_commit+0x5aa
Nov 2 13:47:28 kernel: zfs_sync() at zfs_sync+0xa6
Nov 2 13:47:28 kernel: sync_fsync() at sync_fsync+0x13a
Nov 2 13:47:28 kernel: VOP_F
Nov 2 13:47:28 kernel: SYNC_APV() at VOP_FSYNC_APV+0xb5
Nov 2 13:47:28 kernel: sync_vnode() at sync_vnode+0x157
Nov 2 13:47:28 kernel: sched_sync() at sched_sync+0x1d2
Nov 2 13:47:28 kernel: fork_exit() at fork_exit+0x12a
Nov 2 13:47:28 kernel: fork_trampoline() at fork_trampoline+0xe
Nov 2 13:47:28 kernel: --- trap 0, rip = 0, rsp =
0xffffff8017aebd30, rbp = 0 ---
Nov 2 13:47:44 kernel: lock order reversal:
Nov 2 13:47:44 kernel: 1st 0xffffff002325d4c0 zp->z_parent_lock (zp-
>z_parent_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/
opensolaris/uts/common/fs/zfs/zfs_dir.c:379
Nov 2 13:47:44 kernel: 2nd 0xffffff0002914370 zfsvfs->z_hold_mtx[i]
(zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/
contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863
Nov 2 13:47:44 kernel: KDB: stack backtrace:
Nov 2 13:47:44 kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 2 13:47:44 kernel: _witness_debugger() at _witness_debugger+0x2e
Nov 2 13:47:44 kernel: witness_checkorder() at witness_checkorder
+0x81e
Nov 2 13:47:44 kernel: _sx_xlock() at _sx_xlock+0x55
Nov 2 13:47:44 kernel: zfs_zget() at zfs_zget+0x23a
Nov 2 13:47:44 kernel: zfs_dirlook() at zfs_dirlook+0x1fc
Nov 2 13:47:44 kernel: zfs_lookup() at zfs_lookup+0x257
Nov 2 13:47:44 kernel: zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
Nov 2 13:47:44 kernel: VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV
+0xaf
Nov 2 13:47:44 kernel: vfs_cache_lookup() at vfs_cache_lookup+0xf0
Nov 2 13:47:44 kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
Nov 2 13:47:44 kernel: lookup() at lookup+0x2eb
Nov 2 13:47:44 kernel: namei() at namei+0x4a9
Nov 2 13:47:44 kernel: kern_chdir() at kern_chdir+0x78
Nov 2 13:47:44 kernel: syscall() at syscall+0x1d0
Nov 2 13:47:44 kernel: Xfast_syscall() at Xfast_syscall+0xe1
Nov 2 13:47:44 kernel: --- syscall (12, FreeBSD ELF64, chdir), rip =
0x800da8a9c, rsp = 0x7fffffffe8b8, rbp = 0x8010135e0 ---
More information about the freebsd-fs
mailing list