ZFS is auto-mounting in wrong order

[Trond Endrestøl]

On Thu, 18 Apr 2019 16:33-0700, Jim Long wrote:

> I have a system that boots from ZFS.  I recently upgraded it from
> FreeBSD 11.2-STABLE to 12.0-STABLE:
> 
> FreeBSD electron 12.0-STABLE FreeBSD 12.0-STABLE r346293 GENERIC  amd64
> 
> I'm seeing certain directories as empty, when they should have lots of
> content.  The problem can be worked around by unmounting key
> filesystems, and re-mounting them in the proper order.

We are both experiencing some unforeseen side-effects of r345578,
https://svnweb.freebsd.org/base?view=revision&revision=345578.

I first observed this odd behaviour when upgrading from r345628 to 
r346220. I assumed some of the ZFS changes between these revision were 
to blame, i.e. r346126, r346128, and/or r346130, but alas, r346125 
exhibits the same symptoms as r346220.

I mistook this issue for being deterministic, but it isn't. I have 
rebooted my system now running r346269 a couple of times today, and 
sometimes the parallel mounting succeeds and sometimes it doesn't.

In my particular case it's better to mount all filesystems belonging 
to my root pool in sequential order before mounting in parallel the 
remaining filesystems from the other pools.

Afterall, I only have about 135 filesystems to mount.

I will open a PR suggesting parallel mounting be made optional for us 
who don't have thousands of filesystems to be mounted at boot time. 
Maybe doing the root pool before other pools is the way forward.

See the discussion I initiated on -stable@, 
https://lists.freebsd.org/pipermail/freebsd-stable/2019-April/090898.html

I did make one change today. /var/db from the root pool is no longer a 
separate filesystem. Its contents were moved aside to a directory on 
the root filesystem, /var/db1, the canmount property was set to off, 
and the new directory assumed the correct name of /var/db.

For now, I boot my affected system in single-user mode and run my 
remount shell script saved as /remount-filesystems.sh before exiting 
to multiuser mode:

#!/bin/sh

# To be run while in singleuser mode,
# preferably (re)booted directly to singleuser mode.

PATH="/bin:/sbin:/usr/bin:/usr/sbin:/rescue"
export PATH

killall devd
killall local-unbound
killall moused
killall wpa_supplicant

umount /usr/compat/linux/dev/fd
umount /usr/compat/linux/dev
umount /usr/compat/linux/proc
umount /usr/compat/linux/sys

zfs unmount -a

mount -uw /

for fs in `zfs list -Hro canmount,name enterprise_zroot | egrep -v '(^off)|(enterprise_zroot$)|(enterprise_zroot/ROOT)|(enterprise_zroot/do-not-destroy)' | awk '{print $2}'`; do
  zfs mount ${fs}
done

for fs in `zfs list -Hro canmount,name enterprise_zdata | egrep -v '(^off)|(enterprise_zdata$)|(enterprise_zdata/do-not-destroy)' | awk '{print $2}'`; do
  zfs mount ${fs}
done

mount -al

echo "You may now attempt to exit to multiuser mode ..."

# EOF

-- 
Trond.