ZFS is auto-mounting in wrong order

[Jim Long]

Thank you for taking the time to respond, Paul.

On Fri, Apr 19, 2019 at 10:31:26AM -0400, Paul Mather wrote:
> 
> I notice you have an "electron/system/ROOT" ZFS file system, so maybe you  
> are using boot environments, too?  Do you have any ZFS file systems marked  
> "noauto" for the "canmount" property?  It doesn't seem so, but I thought I  
> would double-check.

No, I'm not using any 'noauto'.  My original post shows all filesystems
that are not 'canmount=off'.  Also, my root FS electron/system/ROOT/default 
has no child filesystems, which seems to be what /etc/rc.d/zfsbe
handles.

> > How can I definitively SEE the order in which ZFS will mount
> > filesystems, either in a "dry-run" mode, or have it logged in real time
> > during the boot process?
> 
> The "service -e" command will show you the order in which services will be  
> started.  In my 12-STABLE system, ZFS related startup scripts seem to run  
> in this order:
> 
> % service -e | grep zfs
> /etc/rc.d/zfsbe
> /etc/rc.d/zfs
> /etc/rc.d/zfsd

I don't use zfsd, and the core of what /etc/rc.d/zfs does is 'zfs mount -va', 
so I still have no visibility into how 'zfs mount' is ordering the
filesystem mounts.

But it is definitely 'zfs mount -va' that is doing the wrong thing.

I've fashioned this test script to highlight the observable differences
between a correct, manual mounting, and the incorrect mount order used
by zfs mount -a:

--snip--
#!/usr/bin/env bash

# correct mount ordering:
MANUAL="
        zfs mount electron/data
        zfs mount electron/data/backup
        zfs mount electron/data/backup/jimsdesk
"

# automatic mount ordering:
AUTO="
        zfs mount -va
"


case $1
in
   manual)
        MOUNT="$MANUAL"
        ;;

   auto)
        MOUNT="$AUTO"
        ;;
   *)
        exit
esac


cat << EOF | sh -v
        zfs mount | grep '/data'
        mkdir /data
        find  /data
$MOUNT
        zfs mount | grep '/data'
        find /data | head -3000000 | wc -l
        zfs umount /data/backup/jimsdesk
        zfs umount /data/backup
        zfs umount /data
        rmdir /data
        find  /data
EOF
--snip--

Here's a run of that script using the manual mounting order.  The
mountpoint /data is created fresh at the start.  When the filesystems
are correctly mounted, there are over 3 million files under /data.
After unmounting, /data is empty, and can be rmdir'ed.

--snip--
electron : 12:26:00 /root# ./zfs-mount-test.sh manual
        zfs mount | grep '/data'
        mkdir /data
        find  /data
/data

        zfs mount electron/data
        zfs mount electron/data/backup
        zfs mount electron/data/backup/jimsdesk

        zfs mount | grep '/data'
electron/data                   /data
electron/data/backup            /data/backup
electron/data/backup/jimsdesk   /data/backup/jimsdesk
        find /data | head -3000000 | wc -l
 3000000
        zfs umount /data/backup/jimsdesk
        zfs umount /data/backup
        zfs umount /data
        rmdir /data
        find  /data
find: /data: No such file or directory
--snip--

Here's a run of that script using the automatic mounting order.  When
the filesystems are automatically mounted, fewer than 1.5 million files
are present under /data.  After unmounting everything, /data is not
clean, and rmdir returns an error.

--snip--
electron : 12:26:17 /root# ./zfs-mount-test.sh auto
        zfs mount | grep '/data'
        mkdir /data
        find  /data
/data

        zfs mount -va

        zfs mount | grep '/data'
electron/data                   /data
electron/data/backup            /data/backup
electron/data/backup/jimsdesk   /data/backup/jimsdesk
        find /data | head -3000000 | wc -l
 1481445
        zfs umount /data/backup/jimsdesk
        zfs umount /data/backup
cannot unmount '/data/backup': not a mountpoint
        zfs umount /data
        rmdir /data
rmdir: /data: Directory not empty
        find  /data
/data
/data/backup
--snip--

Is this a known issue?  Should I submit a PR?


Thank you!

Jim