Re: CURRRENT snapshot won't boot due missing ZFS feature

On Sat, Sep 16, 2023 at 5:11 PM Toomas Soome <tsoome@me.com> wrote:

>
>
> > On 16. Sep 2023, at 18:43, Mark Millard <marklmi@yahoo.com> wrote:
> >
> > void <void_at_f-m.fm> wrote on
> > Date: Sat, 16 Sep 2023 12:12:02 UTC :
> >
> >> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote:
> >>
> >>> Yes. The boot loader comes from the host. It must know how to read
> ZFS.
> >>
> >> It knows how to read zfs.
> >
> > I expect Warner was indicating: you have a (efi?) loader that knows
> > how to deal with the features listed in:
> >
> > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
> >
> > being active but not with some new feature(s) listed in:
> >
> > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> >
> > being active.
> >
> > The following are the "read-only-compatibile no" features
> > that are new in openzfs-2.2 compared to openzfs-2.1-freebsd :
> >
> > blake3
> > ednor
> > head_errlog
> > vdev_zaps_v2
> >
> > So any of those being active leads to lack of even read-only
> > activity being compatible. (Although, the loader's subset
> > of the potential overall activity might allow ignoring some
> > specific "read-only-compatibile no" status examples.)
> >
> > For reference:
> >
> > # diff -u99
> /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
> /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> > ---
> /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
> 2021-06-24 20:08:57.206621000 -0700
> > +++
> /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> 2023-06-10 15:59:25.354999000 -0700
> > @@ -1,34 +1,40 @@
> > -# Features supported by OpenZFS 2.1 on FreeBSD
> > +# Features supported by OpenZFS 2.2 on Linux and FreeBSD
> > allocation_classes
> > async_destroy
> > +blake3
> > +block_cloning
> > bookmark_v2
> > bookmark_written
> > bookmarks
> > device_rebuild
> > device_removal
> > draid
> > +edonr
> > embedded_data
> > empty_bpobj
> > enabled_txg
> > encryption
> > extensible_dataset
> > filesystem_limits
> > +head_errlog
> > hole_birth
> > large_blocks
> > large_dnode
> > livelist
> > log_spacemap
> > lz4_compress
> > multi_vdev_crash_dump
> > obsolete_counts
> > project_quota
> > redacted_datasets
> > redaction_bookmarks
> > resilver_defer
> > sha512
> > skein
> > spacemap_histogram
> > spacemap_v2
> > userobj_accounting
> > +vdev_zaps_v2
> > +zilsaxattr
> > zpool_checkpoint
> > zstd_compress
> >
> > (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does
> > not exist yet. Thus were I had the diff look.)
> >
> >> On the host in question, there are many guests,
> >> some with zfs-boot, some not, just file-based.
> >
> > But with what openzfs features active vs. not active
> > in each case?
> >
> >> What the host is not, is zfs-on-root. It boots from ssd (ada0).
> >> The vdevs are on a sas disk array.
> >>
> >>> So either your bootable partitions must not have
> com.klarasystems:vdev_zaps_v2
> >>> in your BEs or you must have a new user boot. I think you can just
> install
> >>> the one from 14, but haven't tried it.
> >>
> >> Can you briefly explain how I'd install the one from 14 please?
> >
> >
> > I do not use bhyve so I do not even know if the
> > context is using the efi loader from a msdosfs
> > vs. not. For efi loaders, copying from one msdosfs
> > with a sufficient vintage to the one with the wrong
> > vintage (replacing) is sufficient.
>
> bhyve in freebsd is traditionally using /boot/userboot.so, I believe.


Yes. We use the *HOSTS* (running FreeBSD 13) /boot/userboot.so to boot the
FreeBSD 14
image. Since we're not using the boot loader from the target image to load
it for bhyve,
the loader we're using has to understand the ZFS dataset that it's booting
off of. FreeBSD
13's userboot.so doesn't support all the bells and whistles that the ZFS
folks have added
to 14.

So, either you have to turn off those features (which I got no clue how to
do in the
normal installer), or you have to update userboot.so to the FreeBSD 14
version (which
I think had a good chance of actually running on FreeBSD 13 since it has no
'system'
references, which are confined to bhyveload).

Warner


> >
> > # find /boot/efi/EFI/ -print
> > /boot/efi/EFI/
> > /boot/efi/EFI/FREEBSD
> > /boot/efi/EFI/FREEBSD/loader.efi
> > /boot/efi/EFI/BOOT
> > /boot/efi/EFI/BOOT/bootaa64.efi
> >
> > There may well be only:
> >
> > EFI/BOOT/bootaa64.efi
> >
> > for all I know.
> >
> > From an amd64 context:
> >
> > # find /boot/efi/EFI/ -print
> > /boot/efi/EFI/
> > /boot/efi/EFI/FREEBSD
> > /boot/efi/EFI/FREEBSD/loader.efi
> > /boot/efi/EFI/BOOT
> > /boot/efi/EFI/BOOT/bootx64.efi
> >
> > There may well be only:
> >
> > EFI/BOOT/bootx64.efi
> >
> > for all I know.
> >
> > (I set things up to have the EFI capitalization
> > so that referencing efi/ vs. EFI/ in my context
> > is unique for the mount point. vs. the msdosfs
> > directory.)
> >
> > ===
> > Mark Millard
> > marklmi at yahoo.com
> >
> >
>
>
>