[Bug 251363] use unionfs as a disk-cache for NFS [feature]

Sat Nov 28 03:48:07 UTC 2020

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251363

--- Comment #12 from Gunther Schadow <raj at gusw.net> ---
While it works in principle, there is a little problem.

Here is on a system which actually has /usr mounted via NFS. Root sets up the
unionfs cache:

# mkdir /var/cache/
# mount -t unionfs -o copypolicy=always /var/cache/usr /usr
# df
Filesystem                                        1K-blocks              Used  
         Avail Capacity  Mounted on
/dev/gpt/rootfs                                      507280            245844  
        220856    53%    /
devfs                                                     1                 1  
             0   100%    /dev
/dev/gpt/varfs                                       507536            158192  
        308744    34%    /var
fdescfs                                                   1                 1  
             0   100%    /dev/fd
procfs                                                    4                 4  
             0   100%    /proc
/usr                                                 507280            245844  
        220856    53%    /.usr
fs-ca0cf63f.efs.us-east-1.amazonaws.com:/  9007199254739968           6639576
9007199248100392     0%    /usr
<above>:/var/cache/usr                    -9007199254234480 -9007199254583824  
        308744   100%    /usr

Now another user just uses which(1)

$ which java
bash: /usr/bin/which: Permission denied
$ ls -l /usr/bin/which
-r-xr-xr-x  1 root  wheel  7816 Oct 23 06:48 /usr/bin/which
$ ls -l /var/cache/usr/bin
total 0

oh, so the copying of the file did not work? Why?

$ find /var/cache/ -ls
 16394        8 drwxr-xr-x    3 root wheel   512 Nov 26 23:12 /var/cache/
 16384        0 lrwxr-xr-x    1 root wheel    18 Oct 31 18:16 /var/cache/pkg ->
/usr/var/cache/pkg
 16407        8 drwxr-xr-x    8 root wheel   512 Nov 28 03:10 /var/cache/usr
 16387        8 drwxr-xr-x    5 root wheel   512 Nov 28 03:07
/var/cache/usr/local
 16410        8 drwxr-xr-x    2 root wheel   512 Oct 31 03:59
/var/cache/usr/local/etc
 16422        8 drwxr-xr-x    2 root wheel   512 Oct 31 03:59
/var/cache/usr/local/share
 16423        8 drwxr-xr-x    2 root wheel   512 Oct 31 03:59
/var/cache/usr/local/lib
 16416        8 drwxr-xr-x    2 root wheel   512 Oct 23 06:50
/var/cache/usr/sbin
 16417        8 drwxr-xr-x    2 root wheel   512 Oct 31 18:20
/var/cache/usr/bin
 16418        8 drwxr-xr-x    4 root wheel   512 Nov 28 03:07
/var/cache/usr/share
 16420        8 drwxr-xr-x    3 root wheel   512 Nov 28 03:07
/var/cache/usr/share/nls
 16421        8 drwxr-xr-x    2 root wheel   512 Oct 23 06:49
/var/cache/usr/share/nls/C
 16431        8 drwxr-xr-x    2 root wheel   512 Oct 23 06:48
/var/cache/usr/share/zoneinfo
 16424        8 drwxr-xr-x    2 root wheel   512 Nov 28 03:10
/var/cache/usr/lib
 16440      104 -r--r--r--    1 root wheel 53048 Oct 23 06:45
/var/cache/usr/lib/libpam.so.6
 16434        8 drwxr-xr-x    2 root wheel   512 Nov 28 03:10
/var/cache/usr/libexec
 16439       40 -r-xr-xr-x    1 root wheel 16576 Oct 23 06:45
/var/cache/usr/libexec/atrun

Could it be that the problem comes when it's not the root user who is trying to
access the unionfs system? Yes, that could be! Because the expectation of
unionfs_copyfile(...) is that the user actually had write permission!

Prove that, as toot:

# which java
/usr/local/bin/java

and now over for the normal user:

$ which java
/usr/local/bin/java

working too now, and indeed it was copied;

$ ls -l /var/cache/usr/bin/which
-r-xr-xr-x  1 root  wheel  7816 Oct 23 06:48 /var/cache/usr/bin/which

So, to fix this, what I need is a way to run unionfs_copyfile(...) under root
permissions, it being inside the kernel I thought that's what it would do
anyway. But apparently it doesn't and apparently this issue doesn't get to be
seen.

But what about standard unionfs use as copy on write. What if the lower layer
has a file with user root, with group write permission, and now I come as the
non-root user, but in the same group, and wanting to write this file.

Let's try.

# mkdir test-lower
# mkdir test-upper
# ls -l test-*
drwxr-xr-x  2 root      wheel    512 Nov 28 03:34 test-lower
drwxr-xr-x  2 root      wheel    512 Nov 28 03:34 test-upper
# echo "I am root" > test-lower/data
# chmod 664 test-lower/data
# mount -t unionfs test-upper test-lower
# df
Filesystem                                       1K-blocks    Used           
Avail Capacity  Mounted on
...
<above>:/tmp/test-upper                            1014560  753136          
220844    77%    /tmp/test-lower

$ cat test-lower/data
I am root
$ echo "But I'm not root" > test-lower/data
bash: test-lower/data: Permission denied

Look at that! But without unionfs I was able to do that! Let's unmount that
unionfs

# umount test-lower

and try again:

$ echo "But I'm not root" > test-lower/data
$ cat test-lower/data
But I'm not root

So, as we can see, even your normal unionfs is not really transparent! The copy
should have been made with the exact same modes and user:group.

The man page says something that is not actually happening:

             copymode = traditional | transparent | masquerade
                     Specifies the way to create a file or a directory in the
                     upper layer automatically when needed.  The traditional
                     mode uses the same way as the old unionfs for backward
                     compatibility, and transparent duplicates the file and
                     directory mode bits and the ownership in the lower layer
                     to the created file in the upper layer.  For behavior of
                     the masquerade mode, see MASQUERADE MODE below.

"transparent duplicates the file and directory mode bits and the ownership in
the lower layer to the created file in the upper layer."

And that is totally not what's happening! Looks like a bug in unionfs to me
now. But one that I'm determined to squash.

-- 
You are receiving this mail because:
You are the assignee for the bug.