diskless/unionfs panics

[Danny Braniss]

hi,
	after much debugging, it seems that the main problem with unionfs is
that if it's called early in the boot process it will panic the kernel:

trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x0
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xffffffff8038e3f5
stack pointer           = 0x10:0xffffffffb1eac7b0
frame pointer           = 0x10:0xffffffffb1eac7e0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 213 (sh)
[thread pid 213 tid 100066 ]
Stopped at      _mtx_lock_flags+0x35:   cmpq    $0x80779d40,0(%rdi)
db> tr
Tracing pid 213 tid 100066 td 0xffffff007b9b1000
_mtx_lock_flags() at _mtx_lock_flags+0x35
exec_map_first_page() at exec_map_first_page+0x60
kern_execve() at kern_execve+0x2a0
execve() at execve+0x5d
syscall() at syscall+0x4ab
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (59, FreeBSD ELF64, execve), rip = 0x80090630c, rsp = 
0x7fffffffcbf8, rbp = 0 ---

doing the unionfs stuff sometime later - after the single user prompt - seems 
do be ok.

another thing, there are some LOR caused by the unionfs & md:
extarct of rc.d/initdiskless:

if [ -e /conf/union ]; then
    if ! sysctl vfs.uniondebug >/dev/null 2>&1; then
	echo Loading unionfs
	kldload unionfs
    fi
    echo Doing UNIONFS
    mount_md 4096 /conf/etc
    chmod 755 /conf/etc
    mount_unionfs /conf/etc /etc
    ls -R /etc > /dev/null
    touch /etc/.sentinel
    md_created_etc=created
fi
...

Loading unionfs
lock order reversal
 1st 0xffffff007c3b00f0 system map (system map) @ /r+d/6.0/src/sys/vm/vm_map.c:
1100
 2nd 0xffffff00623d8620 vm object (standard object) @ 
/r+d/6.0/src/sys/vm/vm_map.c:897
KDB: stack backtrace:
witness_checkorder() at witness_checkorder+0x5f1
_mtx_lock_flags() at _mtx_lock_flags+0x4a
vm_map_insert() at vm_map_insert+0x115
vm_map_find() at vm_map_find+0x9d
link_elf_load_file() at link_elf_load_file+0x811
linker_load_module() at linker_load_module+0x50e
kldload() at kldload+0xc3
syscall() at syscall+0x4ab
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (304, FreeBSD ELF64, kldload), rip = 0x80067950c, rsp = 
0x7fffffffedf8, rbp = 0x7fffffffee68 ---

Doing UNIONFS
lock order reversal
 1st 0xffffffff80892040 vm page queue mutex (vm page queue mutex) @ 
/r+d/6.0/src/sys/kern/vfs_bio.c:1485
 2nd 0xffffff0061dc98b0 vnode interlock (vnode interlock) @ 
/r+d/6.0/src/sys/kern/vfs_subr.c:1992
KDB: stack backtrace:
witness_checkorder() at witness_checkorder+0x5f1
_mtx_lock_flags() at _mtx_lock_flags+0x4a
vdrop() at vdrop+0x31
vm_page_remove() at vm_page_remove+0x126
vm_page_free_toq() at vm_page_free_toq+0x61
vm_page_free() at vm_page_free+0x1e
vfs_vmio_release() at vfs_vmio_release+0x18f
brelse() at brelse+0x792
ffs_mount() at ffs_mount+0x121b
vfs_donmount() at vfs_donmount+0xaa3
kernel_mount() at kernel_mount+0xaf
ffs_cmount() at ffs_cmount+0x92
mount() at mount+0x132
syscall() at syscall+0x1fb
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (21, FreeBSD ELF64, mount), rip = 0x80067e68c, rsp = 
0x7fffffffdf98, rbp = 0x7fffffffea58 ---