ZFS is auto-mounting in wrong order

Jim Long freebsd-questions at umpquanet.com
Thu Apr 18 23:37:27 UTC 2019

I have a system that boots from ZFS.  I recently upgraded it from
FreeBSD 11.2-STABLE to 12.0-STABLE:

FreeBSD electron 12.0-STABLE FreeBSD 12.0-STABLE r346293 GENERIC  amd64

I'm seeing certain directories as empty, when they should have lots of
content.  The problem can be worked around by unmounting key
filesystems, and re-mounting them in the proper order.

Troubleshooting leads me to believe that the ZFS system is not mounting
them in the proper order, but I don't know of any way to either log the
order that they are mounted in at boot time, nor to inspect on a
hypothetical "dry-run" basis what ZFS thinks the correct mount order is.

I did not have this problem prior to the 11 -> 12 upgrade, if that

Here are my auto-mountable filesystems:

# zfs get -rt filesystem canmount | grep -vw off
NAME                                                         PROPERTY  VALUE     SOURCE
electron                                                     canmount  on        default
electron/data                                                canmount  on        local
electron/data/backup                                         canmount  on        local
electron/data/backup/jimsdesk                                canmount  on        default
electron/system                                              canmount  on        default
electron/system/ROOT                                         canmount  on        default
electron/system/ROOT/default                                 canmount  on        local
electron/system/home                                         canmount  on        default
electron/system/tmp                                          canmount  on        default
electron/system/usr                                          canmount  on        default
electron/system/usr/local                                    canmount  on        default
electron/system/var                                          canmount  on        default

And their mountpoints:

# zfs get -Hrt filesystem canmount | grep -vw off | while read FS b; do echo $FS; done | xargs zfs list
NAME                            USED  AVAIL  REFER  MOUNTPOINT
electron                       80.0T   403G   201K  none
electron/data                  46.4T  34.0T   238K  /data
electron/data/backup           6.67T  34.0T  4.81T  /data/backup
electron/data/backup/jimsdesk   267G  34.0T   267G  /data/backup/jimsdesk
electron/system                13.4G   403G   201K  none
electron/system/ROOT            560M   403G   201K  none
electron/system/ROOT/default    373M   403G   279M  /
electron/system/home            292K   403G   292K  /home
electron/system/tmp            15.5M   403G  13.8M  /tmp
electron/system/usr            12.3G   403G  10.6G  /usr
electron/system/usr/local      1007M   403G   569M  /usr/local
electron/system/var             555M   403G   516M  /var

Let's look at electron/data and its children, although I suspect the
same problem is happening with electron/system/usr and its child

None of the electron/data tree is currently mounted:

# zfs mount
electron/system/ROOT/default    /
electron/system/tmp             /tmp
electron/system/var             /var
electron/system/home            /home
electron/system/usr             /usr
electron/system/usr/local       /usr/local

Since they all mount under /data, the /data directory is currently empty:

# find /data -ls
  1587        1 drwxr-xr-x    2 root                             wheel                                   2 Apr 18 15:33 /data

Now let's mount the remaining auto-mount filesystems:

# zfs mount -a; zfs mount
electron/system/ROOT/default    /
electron/system/tmp             /tmp
electron/system/var             /var
electron/system/home            /home
electron/system/usr             /usr
electron/system/usr/local       /usr/local
electron/data                   /data
electron/data/backup            /data/backup
electron/data/backup/jimsdesk   /data/backup/jimsdesk

Inspect /data and /data/backup:

# ls /data
backup/              slide/         xqueue/        spare/
vince/               mendel/        rain/
# ls /data/backup

There should be a lot more stuff in /data/backup than that.  It's only showing
the mountpoint for electron/data/backup/jimsdesk.  Here's the work-around to
get it to show up:

# zfs umount electron/data/backup/jimsdesk
# zfs umount electron/data/backup
# zfs  mount electron/data/backup
# find /data/backup -maxdepth 1

So clearly the filesystem contents are still there, they were just invisible before.

Since /data is a mountpoint for electron/data, we would expect it to be
empty, and indeed we verified above that it WAS empty, prior to doing
the 'zfs mount -a'.  But instead I find:

# zfs umount electron/data/backup
# zfs umount electron/data
# find /data
# df -m /data/backup
Filesystem                   1M-blocks Used  Avail Capacity  Mounted on
electron/system/ROOT/default    413020  278 412742     0%    /

/data/backup should not exist in electron/system/ROOT/default.  So I
rmdir'ed it and rebooted.  After the reboot, and after unmounting
electron/data/backup/jimsdesk, electron/data/backup and electron/data it
again was present.

This seems like strong evidence that /data/backup is getting mounted
before /data gets mounted, and ZFS is auto-creating the mountpoint

How can I definitively SEE the order in which ZFS will mount
filesystems, either in a "dry-run" mode, or have it logged in real time
during the boot process?

Even more to the point, how can I ensure that /data gets mounted before
/data/backup gets mounted before /data/backup/jimsdesk, and likewise
/usr gets mounted before /usr/local?

Thank you!


More information about the freebsd-questions mailing list