Expensive timeout(9) function: 0xffffffff808b5480(0xffffffff8180a708) 0.021493024 s
Expensive timeout(9) function: 0xffffffff80c98620(0) 3.963288769 s
lock order reversal:
 1st 0xfffff8002d4dc068 zfs (zfs) @ /asp/git/src/sys/kern/vfs_mount.c:1224
 2nd 0xfffff8002d4db7b8 zfs_gfs (zfs_gfs) @ /asp/git/src/sys/cddl/contrib/opensolaris/uts/common/fs/gfs.c:494
stack backtrace:
#0 0xffffffff80a77150 at witness_debugger+0x70
#1 0xffffffff80a77051 at witness_checkorder+0xe71
#2 0xffffffff809f8607 at __lockmgr_args+0xd47
#3 0xffffffff80abf06c at vop_stdlock+0x3c
#4 0xffffffff80f76cb0 at VOP_LOCK1_APV+0x100
#5 0xffffffff80ae1082 at _vn_lock+0xc2
#6 0xffffffff82069b23 at gfs_file_create+0x73
#7 0xffffffff82069bcd at gfs_dir_create+0x1d
#8 0xffffffff821309e7 at zfsctl_mknode_snapdir+0x47
#9 0xffffffff8206a145 at gfs_dir_lookup+0x185
#10 0xffffffff8206a62d at gfs_vop_lookup+0x1d
#11 0xffffffff8212fa05 at zfsctl_root_lookup+0xf5
#12 0xffffffff821308a3 at zfsctl_umount_snapshots+0x83
#13 0xffffffff8214958b at zfs_umount+0x7b
#14 0xffffffff80ac8c10 at dounmount+0x530
#15 0xffffffff80ac864d at sys_unmount+0x35d
#16 0xffffffff80e2b519 at amd64_syscall+0x2f9
#17 0xffffffff80e0acdb at Xfast_syscall+0xfb
lock order reversal:
 1st 0xfffff8002d5978e0 filedesc structure (filedesc structure) @ /asp/git/src/sys/kern/kern_descrip.c:1231
 2nd 0xfffff8014526f7b8 zfs (zfs) @ /asp/git/src/sys/kern/vfs_subr.c:4752
stack backtrace:
#0 0xffffffff80a77150 at witness_debugger+0x70
#1 0xffffffff80a77051 at witness_checkorder+0xe71
#2 0xffffffff809f8607 at __lockmgr_args+0xd47
#3 0xffffffff80abf06c at vop_stdlock+0x3c
#4 0xffffffff80f76cb0 at VOP_LOCK1_APV+0x100
#5 0xffffffff80ae1082 at _vn_lock+0xc2
#6 0xffffffff809daffe at knlist_remove_kq+0x7e
#7 0xffffffff80ad7c58 at filt_vfsdetach+0x28
#8 0xffffffff809db8d7 at knote_fdclose+0xc7
#9 0xffffffff809cfb85 at closefp+0x65
#10 0xffffffff80e2b519 at amd64_syscall+0x2f9
#11 0xffffffff80e0acdb at Xfast_syscall+0xfb
pid 37436 (gvfsd), uid 1001: exited on signal 5
lock order reversal:
 1st 0xfffff8001b0f42d8 syncer (syncer) @ /asp/git/src/sys/kern/vfs_subr.c:1983
 2nd 0xfffff80145b7d068 zfs (zfs) @ /asp/git/src/sys/kern/vfs_subr.c:2435
stack backtrace:
#0 0xffffffff80a77150 at witness_debugger+0x70
#1 0xffffffff80a77051 at witness_checkorder+0xe71
#2 0xffffffff809f8607 at __lockmgr_args+0xd47
#3 0xffffffff80abf06c at vop_stdlock+0x3c
#4 0xffffffff80f76cb0 at VOP_LOCK1_APV+0x100
#5 0xffffffff80ae1082 at _vn_lock+0xc2
#6 0xffffffff80ad07d6 at vget+0x146
#7 0xffffffff80ad2c96 at vfs_msync+0x66
#8 0xffffffff80ad7916 at sync_fsync+0xc6
#9 0xffffffff80f759c5 at VOP_FSYNC_APV+0x115
#10 0xffffffff80ad5c74 at sched_sync+0x3e4
#11 0xffffffff809e45c4 at fork_exit+0x84
#12 0xffffffff80e0af2e at fork_trampoline+0xe



SYSTEM MEMORY INFORMATION:
mem_wire:        5383413760 (   5134MB) [ 66%] Wired: disabled for paging out
mem_active:  +     90443776 (     86MB) [  1%] Active: recently referenced
mem_inactive:+   2483912704 (   2368MB) [ 30%] Inactive: recently not referenced
mem_cache:   +            0 (      0MB) [  0%] Cached: almost avail. for allocation
mem_free:    +     92983296 (     88MB) [  1%] Free: fully available for allocation
mem_gap_vm:  +        16384 (      0MB) [  0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all:     =   8050769920 (   7677MB) [100%] Total real memory managed
mem_gap_sys: +    232927232 (    222MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    =   8283697152 (   7899MB)        Total real memory available
mem_gap_hw:  +    306237440 (    292MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      =   8589934592 (   8192MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:        6013038592 (   5734MB) [ 70%] Logically used memory
mem_avail:   +   2576896000 (   2457MB) [ 29%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   =   8589934592 (   8192MB) [100%] Logically total memory