kern/156039: nullfs + unionfs do not compose, result in unlocking unlocked lock

[John Giacomoni]

>Number:         156039
>Category:       kern
>Synopsis:       nullfs + unionfs do not compose, result in unlocking unlocked lock
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Mar 29 13:30:11 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     John Giacomoni
>Release:        FreeBSD 7.3 and 7.4
>Organization:
LineRate Systems
>Environment:
FreeBSD  7.4-RELEASE FreeBSD 7.4-RELEASE #0: Fri Feb 18 01:55:22 UTC 2011     root at driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
using unionfs to mount a nullfs mounted directory on top of a 3rd directory causes a panic inside vfs.

I am trying to build an appliance with the ability to easily load different versions at boot time based on read-only boot images.

briefly what I have is a partion with 4 directories
/boot
/images
/persist
/running

1. Mount read-only file backed memory disk image, /images/ver1.img, onto /running
2. mount_nullfs /persist /running/persist
3. mount_unionfs /running/persist/usr /running/usr

the problem manifests at boot (during init) as a unlocking unheld lock and it seems to be while the system is launching getty (parallel access race?).

This problem manifests identically whether the mounting is done by an init_script (loader.conf) or an rc.d script.
>How-To-Repeat:
I created a simplified rc.d to demonstrate the problem.

echo over_enable="YES" >> /etc/rc,conf
mkdir -p /over
mkdir -p /null

then add the following script as /etc/rc.d/over


#!/bin/sh

# PROVIDE: over
# BEFORE: dumpon

. /etc/rc.subr

name="over"
start_cmd="over_start"
stop_cmd=":"

over_start()
{
	#
	# Make sure /over and /null exist on root fs
	#

	echo load_kld nullfs.ko
	load_kld nullfs.ko

	echo mount_nullfs /null /over
	mount_nullfs /null /over

	echo load_kld unionfs.ko
	load_kld unionfs.ko

	echo mount_unionfs -ocopymode=transparent /over /usr
	mount_unionfs -ocopymode=transparent /over /usr
}

load_rc_config $name
run_rc_command "$1"

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: