ZFS is auto-mounting in wrong order
Jim Long
freebsd-questions at umpquanet.com
Thu Apr 18 23:37:27 UTC 2019
I have a system that boots from ZFS. I recently upgraded it from
FreeBSD 11.2-STABLE to 12.0-STABLE:
FreeBSD electron 12.0-STABLE FreeBSD 12.0-STABLE r346293 GENERIC amd64
I'm seeing certain directories as empty, when they should have lots of
content. The problem can be worked around by unmounting key
filesystems, and re-mounting them in the proper order.
Troubleshooting leads me to believe that the ZFS system is not mounting
them in the proper order, but I don't know of any way to either log the
order that they are mounted in at boot time, nor to inspect on a
hypothetical "dry-run" basis what ZFS thinks the correct mount order is.
I did not have this problem prior to the 11 -> 12 upgrade, if that
matters.
Here are my auto-mountable filesystems:
# zfs get -rt filesystem canmount | grep -vw off
NAME PROPERTY VALUE SOURCE
electron canmount on default
electron/data canmount on local
electron/data/backup canmount on local
electron/data/backup/jimsdesk canmount on default
electron/system canmount on default
electron/system/ROOT canmount on default
electron/system/ROOT/default canmount on local
electron/system/home canmount on default
electron/system/tmp canmount on default
electron/system/usr canmount on default
electron/system/usr/local canmount on default
electron/system/var canmount on default
And their mountpoints:
# zfs get -Hrt filesystem canmount | grep -vw off | while read FS b; do echo $FS; done | xargs zfs list
NAME USED AVAIL REFER MOUNTPOINT
electron 80.0T 403G 201K none
electron/data 46.4T 34.0T 238K /data
electron/data/backup 6.67T 34.0T 4.81T /data/backup
electron/data/backup/jimsdesk 267G 34.0T 267G /data/backup/jimsdesk
electron/system 13.4G 403G 201K none
electron/system/ROOT 560M 403G 201K none
electron/system/ROOT/default 373M 403G 279M /
electron/system/home 292K 403G 292K /home
electron/system/tmp 15.5M 403G 13.8M /tmp
electron/system/usr 12.3G 403G 10.6G /usr
electron/system/usr/local 1007M 403G 569M /usr/local
electron/system/var 555M 403G 516M /var
Let's look at electron/data and its children, although I suspect the
same problem is happening with electron/system/usr and its child
electron/system/usr/local.
None of the electron/data tree is currently mounted:
# zfs mount
electron/system/ROOT/default /
electron/system/tmp /tmp
electron/system/var /var
electron/system/home /home
electron/system/usr /usr
electron/system/usr/local /usr/local
Since they all mount under /data, the /data directory is currently empty:
# find /data -ls
1587 1 drwxr-xr-x 2 root wheel 2 Apr 18 15:33 /data
Now let's mount the remaining auto-mount filesystems:
# zfs mount -a; zfs mount
electron/system/ROOT/default /
electron/system/tmp /tmp
electron/system/var /var
electron/system/home /home
electron/system/usr /usr
electron/system/usr/local /usr/local
electron/data /data
electron/data/backup /data/backup
electron/data/backup/jimsdesk /data/backup/jimsdesk
Inspect /data and /data/backup:
# ls /data
backup/ slide/ xqueue/ spare/
vince/ mendel/ rain/
# ls /data/backup
jimsdesk/
There should be a lot more stuff in /data/backup than that. It's only showing
the mountpoint for electron/data/backup/jimsdesk. Here's the work-around to
get it to show up:
# zfs umount electron/data/backup/jimsdesk
# zfs umount electron/data/backup
# zfs mount electron/data/backup
# find /data/backup -maxdepth 1
/data/backup
/data/backup/.zfs
/data/backup/rayleigh.CentOS-7
/data/backup/horton
/data/backup/lannister
/data/backup/rayleigh.archived
/data/backup/Thunder
/data/backup/luna
/data/backup/bioinformatics
/data/backup/shiny
/data/backup/salmon
/data/backup/rayleigh.CentOS-6
/data/backup/stark
/data/backup/hoh2
/data/backup/jimsdesk
/data/backup/love
/data/backup/rayleigh
So clearly the filesystem contents are still there, they were just invisible before.
Since /data is a mountpoint for electron/data, we would expect it to be
empty, and indeed we verified above that it WAS empty, prior to doing
the 'zfs mount -a'. But instead I find:
# zfs umount electron/data/backup
# zfs umount electron/data
# find /data
/data
/data/backup
# df -m /data/backup
Filesystem 1M-blocks Used Avail Capacity Mounted on
electron/system/ROOT/default 413020 278 412742 0% /
/data/backup should not exist in electron/system/ROOT/default. So I
rmdir'ed it and rebooted. After the reboot, and after unmounting
electron/data/backup/jimsdesk, electron/data/backup and electron/data it
again was present.
This seems like strong evidence that /data/backup is getting mounted
before /data gets mounted, and ZFS is auto-creating the mountpoint
directory.
How can I definitively SEE the order in which ZFS will mount
filesystems, either in a "dry-run" mode, or have it logged in real time
during the boot process?
Even more to the point, how can I ensure that /data gets mounted before
/data/backup gets mounted before /data/backup/jimsdesk, and likewise
/usr gets mounted before /usr/local?
Thank you!
Jim
More information about the freebsd-questions
mailing list