kern/161511: Filesystem deadlocks when using multiple unionfs mounts on top of single filesystem

[Garrett Cooper]

>Number:         161511
>Category:       kern
>Synopsis:       Filesystem deadlocks when using multiple unionfs mounts on top of single filesystem
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 12 04:10:09 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Garrett Cooper
>Release:        10-CURRENT
>Organization:
iXsystems, Inc.
>Environment:
FreeBSD streetfighter.ixsystems.com 10.0-CURRENT FreeBSD 10.0-CURRENT #4 r226140M: Tue Oct 11 17:03:51 PDT 2011     root at streetfighter.ixsystems.com:/usr/obj/usr/src/sys/STREETFIGHTER  amd64
>Description:
Mounting a filesystem twice with unionfs like so causes filesystem deadlocks when files are modified on top of unionfs:

#!/bin/sh

mkdir pristine
cat > supfile <<EOF
*default host=cvsup1.freebsd.org
*default base=/FreeBSD/sup
*default prefix=$PWD/FreeBSD
*default release=cvs
*default delete use-rel-suffix
*default compress
EOF
csup supfile
mkdir touched union
mount -t unionfs pristine union
mount -t unionfs touched union
cd union
make buildworld -j4

=====

Any value of -j above -j1 causes filesystem deadlocks; processes like make will block when trying to access directory information -- it's worse for UFS than ZFS because it seems like some structures -- in particular the UFS_DIRHASH -- might be locked indefinitely causing the entire filesystem to become unusable. In either case I've seen the filesystem will prevent rebooting from completing cleanly as there are processes waiting on filesystem resources to become available.

This can also be stimulated via freenas builds like so:

sh build/do_build.sh -U

LORs have also been identified within unionfs around when the deadlock occurs, which might suggest that incorrect locking is preventing unionfs from releasing resources when a [soft] failure occurs. I don't have the LORs right now, but I'm hunting them down in my logs..
>How-To-Repeat:
See decomposed example above.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: