kern/178091: [md/nfs/vfs] md filesystem layering not handled correctly when stacking on top of NFS; works with tempfs

[Garrett Cooper]

>Number:         178091
>Category:       kern
>Synopsis:       [md/nfs/vfs] md filesystem layering not handled correctly when stacking on top of NFS; works with tempfs
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Apr 23 21:40:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Garrett Cooper
>Release:        10-CURRENT
>Organization:
EMC Isilon
>Environment:
FreeBSD mina3010.west.isilon.com 10.0-CURRENT FreeBSD 10.0-CURRENT #1: Sat Apr 20 07:51:00 PDT 2013     root at mina3010.west.isilon.com:/usr/obj/usr/src/sys/ISI-GENERIC  amd64
>Description:
I intentionally layered these filesystems on a build client machine in order to avoid having to rework things with respect to MAKEOBJDIRPREFIX, but it turns out this isn't possible with memory disks mounted on NFS. Example:

Client:

# uname -a
FreeBSD mina3010.west.isilon.com 10.0-CURRENT FreeBSD 10.0-CURRENT #1: Sat Apr 20 07:51:00 PDT 2013     root at mina3010.west.isilon.com:/usr/obj/usr/src/sys/ISI-GENERIC  amd64
# stat -f %d /mnt/onefs.3/obj/
973143811
# stat -f %d /mnt/onefs.3/src/
973143811
# mount | grep /mnt
gran-tourismo:/home/gcooper/svn on /mnt (nfs, asynchronous)
/dev/md42 on /mnt/onefs.3/obj (ufs, local, soft-updates)

This however functions as I would expect with memory disks mounted on ZFS:

Client/Server:

# uname -a
FreeBSD gran-tourismo.west.isilon.com 10.0-CURRENT FreeBSD 10.0-CURRENT #2 r+ba9afe9: Fri Apr 19 20:29:10 PDT 2013     gcooper at gran-tourismo.west.isilon.com:/usr/obj/usr/src/sys/GRAN-TOURISMO  amd64
# mkdir foo; mdmfs -s 100m md4 foo
load: 12.79  cmd: mdmfs 43756 [tx->tx_quiesce_done_cv] 2.29r 0.00u 0.00s 0% 196k
# stat -f %d $PWD/foo/
124
# stat -f %d $PWD
1152244014

There are some VFS layering issues and dot-dot lookup issues with md(4)/NFS that really need to be resolved, because really the device numbers returned for both paths should be different.

tempfs works BTW in place of md(4):

# umount /mnt/onefs.3/obj
# mdconfig -d -u 42
# mount -o size=$(( 100 * 1024 * 1024 )) -t tmpfs tmpfs /mnt/onefs.3/obj
# stat -f %d /mnt/onefs.3/obj/
2264989444
# stat -f %d /mnt/onefs.3/src
973143811
>How-To-Repeat:
1. Mount an NFS share on CURRENT from a CURRENT NFS server.
2. Create a directory on the share.
3. Mount a memory disk on the directory on the share [using mdmfs].
4. Run stat -f %d on the directory on the share and the share itself. The values should be different.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: