zfs_enable vs zfs_load in loader.conf (but neither works)

[J David]

Thanks very much for the info Andriy.

On Fri, Sep 13, 2013 at 9:22 AM, Andriy Gapon <avg at freebsd.org> wrote:
> Another piece of information is that neither mountpoint nor canmount property
> affects ZFS root mounting.

It is mountpoint=legacy that boots on this machine and mountpoint=/
that can't find init, with no other changes.  So clearly under some
obscure edge case, this is not strictly correct.

Did your test include zpool root != filesystem root?  Because as you
describe one possible cause of the problem is mounting the wrong
filesystem as root, one wonders if somehow with the mountpoint=/
setting the zpool root (which has no files at all) is incorrectly
being chosen as fsroot with mountpoint=/ on data/root.  I.e. perhaps
somewhere in the code is looking for "legacy" (or skipping anything
with a mountpoint set) and defaulting back to the zpool root if it is
not found?

Unfortunately there is no way I know of to check and see what the root
filesystem turned out to be after the failure to find init.  That
would reduce speculation about what is happening quite a bit.  Can the
kernel debugger extract this info?

If it matters, the pools in question are fairly complex, not just
throwing everything possible on one drive.  Example:

NAME        STATE     READ WRITE CKSUM
data        ONLINE       0     0     0
 mirror-0  ONLINE       0     0     0
   da0p2   ONLINE       0     0     0
   da4p2   ONLINE       0     0     0
 mirror-1  ONLINE       0     0     0
   da1p2   ONLINE       0     0     0
   da5p2   ONLINE       0     0     0
 mirror-2  ONLINE       0     0     0
   da2p2   ONLINE       0     0     0
   da6p2   ONLINE       0     0     0
 mirror-3  ONLINE       0     0     0
   da3p2   ONLINE       0     0     0
   da7p2   ONLINE       0     0     0
logs
 ada0p2    ONLINE       0     0     0
cache
 ada1p2    ONLINE       0     0     0

The boot order for that system begins at either ada0 or da0, not sure
which without checking the BIOS.

Thanks!