ZFS parallel mounting gone wrong?

Trond Endrestøl Trond.Endrestol at fagskolen.gjovik.no
Wed Apr 17 09:12:42 UTC 2019

On Mon, 15 Apr 2019 15:24+0200, Trond Endrestøl wrote:

> I upgraded a non-critical system running amd64 stable/12 to r346220.
> During multiuser boot not all ZFS filesystems below zroot/usr/local 
> was mounted.

Some more explaining is in order:

This system has two 7 year old pools that complement each other.

/usr/local comes mostly from the zroot pool, but other children comes 
from a zdata pool. The intermediary filesystems have their canmount 
property set to off and mountpoints are specified at the top level 
only. The same goes for other parts of the filesystem hierarchy, such 
as /var/db and /var/spool.

I just upgraded to stable/12 global r346269, local r346268. During 
a routine "zfs mount -av" performed in single user mode, the kernel 
proceeded to mount a child filesystem (enterprise_zdata/var/db/mysql) 
without the parent filesystems being mounted first.

I rebooted back to r345628 from March 28th, and this kernel has no 
problems correctly mounting the ZFS filesystems in parallel. That BE 
used LLVM 7.0.1 from base as its system compiler.

Rebooting into r346220 (April 15th) or r346269 (April 17th) clearly 
shows problems mounting filesystems in the correct order. These BEs 
was compiled using LLVM 8.0.0 from base.

Maybe the system compiler is irrelevant.

The name of the pools might also be a factor, the zdata pool preceedes 
the zroot pool in alphanumerical order.

Maybe there is a bug in the code, or the code breaks when parts of the 
filesystem hierarchy is being built from multiple pools.

Here's an attempt at explaining how this fits together:

zfs list -ro name,canmount,mountpoint enterprise_zroot/usr enterprise_zdata/usr enterprise_zroot/var enterprise_zdata/var
[the list has been slightly edited, moving zdata below zroot and adding an empty line]

NAME                                                   CANMOUNT  MOUNTPOINT
enterprise_zroot/usr                                        off  /usr
enterprise_zroot/usr/compat                                  on  /usr/compat
enterprise_zroot/usr/local                                   on  /usr/local
enterprise_zroot/usr/local/certs                             on  /usr/local/certs
enterprise_zroot/usr/local/etc                               on  /usr/local/etc
enterprise_zroot/usr/local/etc/shellkonfig3                  on  /usr/local/etc/shellkonfig3
enterprise_zroot/usr/local/etc/shellkonfig3/head             on  /usr/local/etc/shellkonfig3/head
enterprise_zroot/usr/local/etc/shellkonfig3/stable-10        on  /usr/local/etc/shellkonfig3/stable-10
enterprise_zroot/usr/local/etc/shellkonfig3/stable-11        on  /usr/local/etc/shellkonfig3/stable-11
enterprise_zroot/usr/local/etc/shellkonfig3/stable-8         on  /usr/local/etc/shellkonfig3/stable-8
enterprise_zroot/usr/local/etc/shellkonfig3/stable-9         on  /usr/local/etc/shellkonfig3/stable-9
enterprise_zroot/usr/local/info                              on  /usr/local/info
enterprise_zroot/usr/local/var                               on  /usr/local/var
enterprise_zroot/usr/obj                                     on  /usr/obj
enterprise_zroot/usr/ports                                   on  /usr/ports
enterprise_zroot/usr/ports/distfiles                         on  /usr/ports/distfiles
enterprise_zroot/usr/ports/local                            off  /usr/ports/local
enterprise_zroot/usr/ports/packages                          on  /usr/ports/packages
enterprise_zroot/usr/ports/workdirs                          on  /usr/ports/workdirs
enterprise_zroot/usr/src                                     on  /usr/src
enterprise_zdata/usr                                        off  /usr
enterprise_zdata/usr/local                                  off  /usr/local
enterprise_zdata/usr/local/moodledata                        on  /usr/local/moodledata
enterprise_zdata/usr/local/pgsql                             on  /usr/local/pgsql
enterprise_zdata/usr/local/restaurering                      on  /usr/local/restaurering
enterprise_zdata/usr/local/www                               on  /usr/local/www
enterprise_zdata/usr/local/www/moodle                        on  /usr/local/www/moodle

enterprise_zroot/var                                        off  /var
enterprise_zroot/var/Named                                   on  /var/Named
enterprise_zroot/var/account                                 on  /var/account
enterprise_zroot/var/audit                                   on  /var/audit
enterprise_zroot/var/cache                                  off  /var/cache
enterprise_zroot/var/cache/ccache                            on  /var/cache/ccache
enterprise_zroot/var/cache/synth                             on  /var/cache/synth
enterprise_zroot/var/crash                                   on  /var/crash
enterprise_zroot/var/db                                      on  /var/db
enterprise_zroot/var/db/darkstat                             on  /var/db/darkstat
enterprise_zroot/var/db/dkim                                 on  /var/db/dkim
enterprise_zroot/var/db/etcupdate                            on  /var/db/etcupdate
enterprise_zroot/var/db/hyperv                               on  /var/db/hyperv
enterprise_zroot/var/db/ntp                                  on  /var/db/ntp
enterprise_zroot/var/db/pkg                                  on  /var/db/pkg
enterprise_zroot/var/db/ports                                on  /var/db/ports
enterprise_zroot/var/db/sup                                  on  /var/db/sup
enterprise_zroot/var/empty                                   on  /var/empty
enterprise_zroot/var/log                                     on  /var/log
enterprise_zroot/var/mail                                    on  /var/mail
enterprise_zroot/var/munin                                   on  /var/munin
enterprise_zroot/var/run                                     on  /var/run
enterprise_zroot/var/spool                                   on  /var/spool
enterprise_zroot/var/spool/cvsup                             on  /var/spool/cvsup
enterprise_zroot/var/synth                                   on  /var/synth
enterprise_zroot/var/synth/builders                          on  /var/synth/builders
enterprise_zroot/var/synth/live_packages                     on  /var/synth/live_packages
enterprise_zroot/var/tmp                                     on  /var/tmp
enterprise_zroot/var/unbound                                 on  /var/unbound
enterprise_zdata/var                                        off  /var
enterprise_zdata/var/db                                     off  /var/db
enterprise_zdata/var/db/mysql                                on  /var/db/mysql
enterprise_zdata/var/db/mysql_secure                         on  /var/db/mysql_secure
enterprise_zdata/var/db/mysql_tmpdir                         on  /var/db/mysql_tmpdir
enterprise_zdata/var/db/postgres                             on  /var/db/postgres
enterprise_zdata/var/db/postgres/data11                      on  /var/db/postgres/data11
enterprise_zdata/var/db/postgres/data11/base                 on  /var/db/postgres/data11/base
enterprise_zdata/var/db/postgres/data11/pg_wal               on  /var/db/postgres/data11/pg_wal
enterprise_zdata/var/db/postgres/data96                      on  /var/db/postgres/data96
enterprise_zdata/var/db/postgres/data96/base                 on  /var/db/postgres/data96/base
enterprise_zdata/var/db/postgres/data96/pg_xlog              on  /var/db/postgres/data96/pg_xlog
enterprise_zdata/var/db/prometheus                           on  /var/db/prometheus
enterprise_zdata/var/db/prometheus/data                      on  /var/db/prometheus/data
enterprise_zdata/var/db/prometheus/data/wal                  on  /var/db/prometheus/data/wal
enterprise_zdata/var/spool                                  off  /var/spool
enterprise_zdata/var/spool/bareos                            on  /var/spool/bareos
enterprise_zdata/var/spool/ftp                               on  /var/spool/ftp

Using this remount script in singleuser mode, brings order to chaos:


# To be run while in singleuser mode,
# preferably (re)booted directly to singleuser mode.

export PATH

killall devd
killall moused

umount /usr/compat/linux/dev/fd
umount /usr/compat/linux/dev
umount /usr/compat/linux/proc
umount /usr/compat/linux/sys

zfs unmount -a

for fs in `zfs list -Hro canmount,name enterprise_zroot | grep -v '^off' | grep -v 'enterprise_zroot$' | grep -v 'enterprise_zroot/ROOT' | grep -v 'enterprise_zroot/do-not-destroy' | awk '{print $2}'`; do
  zfs mount ${fs}

for fs in `zfs list -Hro canmount,name enterprise_zdata | grep -v '^off' | grep -v 'enterprise_zdata$'                                   | grep -v 'enterprise_zdata/do-not-destroy' | awk '{print $2}'`; do
  zfs mount ${fs}

mount -al

echo "You may now attempt to exit to multiuser mode ..."


-- Trond.

More information about the freebsd-stable mailing list