Re: CURRRENT snapshot won't boot due missing ZFS feature

On Sat, 16 Sep 2023 08:43:49 -0700
Mark Millard <marklmi@yahoo.com> wrote:

> void <void_at_f-m.fm> wrote on
> Date: Sat, 16 Sep 2023 12:12:02 UTC :
> 
> > On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote:
> > 
> > >Yes. The boot loader comes from the host. It must know how to read ZFS. 
> > 
> > It knows how to read zfs.
> 
> I expect Warner was indicating: you have a (efi?) loader that knows
> how to deal with the features listed in:
> 
> sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
> 
> being active but not with some new feature(s) listed in:
> 
> sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> 
> being active.
> 
> The following are the "read-only-compatibile no" features
> that are new in openzfs-2.2 compared to openzfs-2.1-freebsd :
> 
> blake3
> ednor
> head_errlog
> vdev_zaps_v2
> 
> So any of those being active leads to lack of even read-only
> activity being compatible. (Although, the loader's subset
> of the potential overall activity might allow ignoring some
> specific "read-only-compatibile no" status examples.)
> 
> For reference:
> 
> # diff -u99 /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> --- /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd 2021-06-24 20:08:57.206621000 -0700
> +++ /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 2023-06-10 15:59:25.354999000 -0700
> @@ -1,34 +1,40 @@
> -# Features supported by OpenZFS 2.1 on FreeBSD
> +# Features supported by OpenZFS 2.2 on Linux and FreeBSD
>  allocation_classes
>  async_destroy
> +blake3
> +block_cloning
>  bookmark_v2
>  bookmark_written
>  bookmarks
>  device_rebuild
>  device_removal
>  draid
> +edonr
>  embedded_data
>  empty_bpobj
>  enabled_txg
>  encryption
>  extensible_dataset
>  filesystem_limits
> +head_errlog
>  hole_birth
>  large_blocks
>  large_dnode
>  livelist
>  log_spacemap
>  lz4_compress
>  multi_vdev_crash_dump
>  obsolete_counts
>  project_quota
>  redacted_datasets
>  redaction_bookmarks
>  resilver_defer
>  sha512
>  skein
>  spacemap_histogram
>  spacemap_v2
>  userobj_accounting
> +vdev_zaps_v2
> +zilsaxattr
>  zpool_checkpoint
>  zstd_compress
> 
> (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does
> not exist yet. Thus were I had the diff look.)

It may be because it's not yet listed here, thus not installed.

  /usr/src/cddl/share/zfs/compatibility.d/Makefile

> > On the host in question, there are many guests,
> > some with zfs-boot, some not, just file-based.
> 
> But with what openzfs features active vs. not active
> in each case?
> 
> > What the host is not, is zfs-on-root. It boots from ssd (ada0).
> > The vdevs are on a sas disk array.
> > 
> > >So either your bootable partitions must not have com.klarasystems:vdev_zaps_v2
> > >in your BEs or you must have a new user boot. I think you can just install
> > >the one from 14, but haven't tried it.
> > 
> > Can you briefly explain how I'd install the one from 14 please?
> 
> 
> I do not use bhyve so I do not even know if the
> context is using the efi loader from a msdosfs
> vs. not. For efi loaders, copying from one msdosfs
> with a sufficient vintage to the one with the wrong
> vintage (replacing) is sufficient.
> 
> For reference (from an aarch64 context):
> 
> # find /boot/efi/EFI/ -print
> /boot/efi/EFI/
> /boot/efi/EFI/FREEBSD
> /boot/efi/EFI/FREEBSD/loader.efi
> /boot/efi/EFI/BOOT
> /boot/efi/EFI/BOOT/bootaa64.efi
> 
> There may well be only:
> 
> EFI/BOOT/bootaa64.efi
> 
> for all I know.
> 
> >From an amd64 context:
> 
> # find /boot/efi/EFI/ -print
> /boot/efi/EFI/
> /boot/efi/EFI/FREEBSD
> /boot/efi/EFI/FREEBSD/loader.efi
> /boot/efi/EFI/BOOT
> /boot/efi/EFI/BOOT/bootx64.efi
> 
> There may well be only:
> 
> EFI/BOOT/bootx64.efi
> 
> for all I know.
> 
> (I set things up to have the EFI capitalization
> so that referencing efi/ vs. EFI/ in my context
> is unique for the mount point. vs. the msdosfs
> directory.)
> 
> ===
> Mark Millard
> marklmi at yahoo.com

-- 
Tomoaki AOKI    <junchoon@dec.sakura.ne.jp>