ZFS is auto-mounting in wrong order

[Paul Mather]

Jim Long <freebsd-questions at umpquanet.com> wrote:

> I have a system that boots from ZFS.  I recently upgraded it from
> FreeBSD 11.2-STABLE to 12.0-STABLE:
>
> FreeBSD electron 12.0-STABLE FreeBSD 12.0-STABLE r346293 GENERIC  amd64
>
> I'm seeing certain directories as empty, when they should have lots of
> content.  The problem can be worked around by unmounting key
> filesystems, and re-mounting them in the proper order.
>
> Troubleshooting leads me to believe that the ZFS system is not mounting
> them in the proper order, but I don't know of any way to either log the
> order that they are mounted in at boot time, nor to inspect on a
> hypothetical "dry-run" basis what ZFS thinks the correct mount order is.
>
> I did not have this problem prior to the 11 -> 12 upgrade, if that
> matters.


I encountered this problem in upgrading from 11-STABLE -> 12-STABLE.  Under  
11-STABLE and prior, when upgrading in single-user mode I would use "zfs  
mount -a" to mount my ZFS file systems.  Under 12-STABLE, this resulted in  
many "empty directories" as you mention.  It caused the subsequent "make  
installworld" to fail.

In my case, I discovered it was the way my boot environment had been set up  
that was causing this.  The file systems under my boot environment "root"  
(pool/ROOT) were all marked as "noauto".  Under 11-STABLE, "zfs mount -a"  
would successfully mount these file systems (even though the man page said  
it shouldn't).  Under 12-STABLE, "zfs mount -a" acts in accordance with the  
man page and does not mount these file systems in response to "zfs mount  
-a".

My workaround was now to mount ZFS file systems as the system does: first  
execute /etc/rc.d/zfsbe and then /etc/rc.d/zfs, i.e., "/etc/rc.d/zfsbe  
start ; /etc/rc.d/zfs start".  That correctly mounts the active boot  
environment and then the rest of the ZFS file systems on top of it.

I notice you have an "electron/system/ROOT" ZFS file system, so maybe you  
are using boot environments, too?  Do you have any ZFS file systems marked  
"noauto" for the "canmount" property?  It doesn't seem so, but I thought I  
would double-check.


> This seems like strong evidence that /data/backup is getting mounted
> before /data gets mounted, and ZFS is auto-creating the mountpoint
> directory.
>
> How can I definitively SEE the order in which ZFS will mount
> filesystems, either in a "dry-run" mode, or have it logged in real time
> during the boot process?


The "service -e" command will show you the order in which services will be  
started.  In my 12-STABLE system, ZFS related startup scripts seem to run  
in this order:

% service -e | grep zfs
/etc/rc.d/zfsbe
/etc/rc.d/zfs
/etc/rc.d/zfsd


The zfsbe service handles mounting of the active boot environment.  The zfs  
service mounts the rest of the ZFS filesystems (the non-"noauto" ones, that  
is).  (The zfsd service runs the ZFS fault management daemon, driven by  
devd events.)

You could run these scripts manually via "/bin/sh -x" to trace what they  
are doing.

I hope this helps.

Cheers,

Paul.