possible hang in unionfs

[Volodymyr Kostyrko]

Hello.

I'm supporting servers that share some projects, and I want those 
projects to be contamined and separated. I'm using this scheme:

/bin /var/jail/project/bin nullfs ro,late 0 0
/tmp /var/jail/project/tmp unionfs ro,late,below 0 0
/var /var/jail/project/var unionfs ro,late,below 0 0
/usr /var/jail/project/usr nullfs ro,late 0 0
/usr/local /var/jail/project/usr/local nullfs ro,late 0 0
/lib /var/jail/project/lib nullfs ro,late 0 0

My base system is ZFS hence mounting /usr and /usr/local as they are 
different fs. And because I want to access postgres/mysql/redis through 
socket I need tmp from the main container. And the sites want write 
access to tmp too.

Mostly everything works, but sometimes the machine gets locked. 
Everything stops and I can't even login. I can see login process is 
waiting in [zfs].

I rebuild everything with DEBUG and now I see this:

Sep 14 11:19:36 slylandro kernel: lock order reversal:
Sep 14 11:19:36 slylandro kernel: 1st 0xfffffe002e41b878 tmpfs (tmpfs) @ 
/usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:1907
Sep 14 11:19:36 slylandro kernel: 2nd 0xfffffe019b06c680 zfs (zfs) @ 
/usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:1938
Sep 14 11:19:36 slylandro kernel: KDB: stack backtrace:
Sep 14 11:19:36 slylandro kernel: db_trace_self_wrapper() at 
db_trace_self_wrapper+0x2b/frame 0xffffff9f8492d2d0
Sep 14 11:19:36 slylandro kernel: kdb_backtrace() at 
kdb_backtrace+0x39/frame 0xffffff9f8492d380
Sep 14 11:19:36 slylandro kernel: witness_checkorder() at 
witness_checkorder+0xc0a/frame 0xffffff9f8492d400
Sep 14 11:19:36 slylandro kernel: __lockmgr_args() at 
__lockmgr_args+0x744/frame 0xffffff9f8492d520
Sep 14 11:19:36 slylandro kernel: vop_stdlock() at 
vop_stdlock+0x3c/frame 0xffffff9f8492d540
Sep 14 11:19:36 slylandro kernel: VOP_LOCK1_APV() at 
VOP_LOCK1_APV+0xbe/frame 0xffffff9f8492d570
Sep 14 11:19:36 slylandro kernel: unionfs_lock() at 
unionfs_lock+0x4c6/frame 0xffffff9f8492d600
Sep 14 11:19:36 slylandro kernel: VOP_LOCK1_APV() at 
VOP_LOCK1_APV+0xbe/frame 0xffffff9f8492d630
Sep 14 11:19:36 slylandro kernel: _vn_lock() at _vn_lock+0x54/frame 
0xffffff9f8492d690
Sep 14 11:19:36 slylandro kernel: unionfs_root() at 
unionfs_root+0x43/frame 0xffffff9f8492d6c0
Sep 14 11:19:36 slylandro kernel: lookup() at lookup+0x99d/frame 
0xffffff9f8492d750
Sep 14 11:19:36 slylandro kernel: namei() at namei+0x589/frame 
0xffffff9f8492d800
Sep 14 11:19:36 slylandro kernel: unp_connect() at 
unp_connect+0x1b3/frame 0xffffff9f8492da60
Sep 14 11:19:36 slylandro kernel: uipc_connect() at 
uipc_connect+0x47/frame 0xffffff9f8492da90
Sep 14 11:19:36 slylandro kernel: kern_connect() at 
kern_connect+0xdb/frame 0xffffff9f8492dae0
Sep 14 11:19:36 slylandro kernel: sys_connect() at 
sys_connect+0x96/frame 0xffffff9f8492db20
Sep 14 11:19:36 slylandro kernel: amd64_syscall() at 
amd64_syscall+0x259/frame 0xffffff9f8492dc30
Sep 14 11:19:36 slylandro kernel: Xfast_syscall() at 
Xfast_syscall+0xfb/frame 0xffffff9f8492dc30
Sep 14 11:19:36 slylandro kernel: --- syscall (98, FreeBSD ELF64, 
sys_connect), rip = 0x801f4d8ac, rsp = 0x7fffffff6c18, rbp = 
0x7fffffff6c40 ---
Sep 14 11:19:39 slylandro kernel: lock order reversal:
Sep 14 11:19:39 slylandro kernel: 1st 0xfffffe0089d87878 zfs (zfs) @ 
/usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:312
Sep 14 11:19:39 slylandro kernel: 2nd 0xfffffe0280cc9680 unionfs 
(unionfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_subr.c:366
Sep 14 11:19:39 slylandro kernel: 3rd 0xfffffe002e3ec098 zfs (zfs) @ 
/usr/src/sys/kern/vfs_subr.c:2335
Sep 14 11:19:39 slylandro kernel: KDB: stack backtrace:
Sep 14 11:19:39 slylandro kernel: db_trace_self_wrapper() at 
db_trace_self_wrapper+0x2b/frame 0xffffff9f835fe120
Sep 14 11:19:39 slylandro kernel: kdb_backtrace() at 
kdb_backtrace+0x39/frame 0xffffff9f835fe1d0
Sep 14 11:19:39 slylandro kernel: witness_checkorder() at 
witness_checkorder+0xc0a/frame 0xffffff9f835fe250
Sep 14 11:19:39 slylandro kernel: __lockmgr_args() at 
__lockmgr_args+0x744/frame 0xffffff9f835fe370
Sep 14 11:19:39 slylandro kernel: vop_stdlock() at 
vop_stdlock+0x3c/frame 0xffffff9f835fe390
Sep 14 11:19:39 slylandro kernel: VOP_LOCK1_APV() at 
VOP_LOCK1_APV+0xbe/frame 0xffffff9f835fe3c0
Sep 14 11:19:39 slylandro kernel: _vn_lock() at _vn_lock+0x54/frame 
0xffffff9f835fe420
Sep 14 11:19:39 slylandro kernel: vputx() at vputx+0x21b/frame 
0xffffff9f835fe480
Sep 14 11:19:39 slylandro kernel: unionfs_noderem() at 
unionfs_noderem+0x1cf/frame 0xffffff9f835fe4f0
Sep 14 11:19:39 slylandro kernel: unionfs_reclaim() at 
unionfs_reclaim+0x14/frame 0xffffff9f835fe500
Sep 14 11:19:39 slylandro kernel: VOP_RECLAIM_APV() at 
VOP_RECLAIM_APV+0xb9/frame 0xffffff9f835fe530
Sep 14 11:19:39 slylandro kernel: vgonel() at vgonel+0x16e/frame 
0xffffff9f835fe5a0
Sep 14 11:19:39 slylandro kernel: vrecycle() at vrecycle+0x3e/frame 
0xffffff9f835fe5d0
Sep 14 11:19:39 slylandro kernel: unionfs_inactive() at 
unionfs_inactive+0x23/frame 0xffffff9f835fe5e0
Sep 14 11:19:39 slylandro kernel: VOP_INACTIVE_APV() at 
VOP_INACTIVE_APV+0xb9/frame 0xffffff9f835fe610
Sep 14 11:19:39 slylandro kernel: vinactive() at vinactive+0xc6/frame 
0xffffff9f835fe660
Sep 14 11:19:39 slylandro kernel: vputx() at vputx+0x263/frame 
0xffffff9f835fe6c0
Sep 14 11:19:39 slylandro kernel: lookup() at lookup+0xd2c/frame 
0xffffff9f835fe750
Sep 14 11:19:39 slylandro kernel: namei() at namei+0x589/frame 
0xffffff9f835fe800
Sep 14 11:19:39 slylandro kernel: unp_connect() at 
unp_connect+0x1b3/frame 0xffffff9f835fea60
Sep 14 11:19:39 slylandro kernel: uipc_connect() at 
uipc_connect+0x47/frame 0xffffff9f835fea90
Sep 14 11:19:39 slylandro kernel: kern_connect() at 
kern_connect+0xdb/frame 0xffffff9f835feae0
Sep 14 11:19:39 slylandro kernel: sys_connect() at 
sys_connect+0x96/frame 0xffffff9f835feb20
Sep 14 11:19:39 slylandro kernel: amd64_syscall() at 
amd64_syscall+0x259/frame 0xffffff9f835fec30
Sep 14 11:19:39 slylandro kernel: Xfast_syscall() at 
Xfast_syscall+0xfb/frame 0xffffff9f835fec30
Sep 14 11:19:39 slylandro kernel: --- syscall (98, FreeBSD ELF64, 
sys_connect), rip = 0x801f4d8ac, rsp = 0x7fffffff7628, rbp = 
0x7fffffff7650 ---

The machine is on 9.2-RC4.

Has everyone faced something like this? Is there any other way to share 
host and slave containers filesystems?

-- 
Sphinx of black quartz, judge my vow.