From nobody Sat Sep 16 16:22:20 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RnxDC6Krgz4tg9r for ; Sat, 16 Sep 2023 16:22:35 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RnxDC36lnz4n42 for ; Sat, 16 Sep 2023 16:22:35 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x530.google.com with SMTP id 4fb4d7f45d1cf-52683da3f5cso3937083a12.3 for ; Sat, 16 Sep 2023 09:22:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1694881351; x=1695486151; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=a+yBHB7NxTpsynDoEBqNFk6YAWtINqaaQmccUbmFZv8=; b=bQP9BRreat9mdXOdBmhMagJKhZ3xs6TN5I9Z7htU5vlMAjlygBnqjjZccOSoWK/8HF JE20Zax+4y1G/oeBiX6HuzPry8bk2cxzy2sN9PO2mz4MhYv7LZ8EN6ue8wUitmDXfIv7 iwdgXKyHmT98nNGwNB86OGG7jUD25/+eIJKE2XdtVQE8mWqrBuy1BXSyk7SvcGZLy1/F hWrru/juBeE/4FUcOxTZrbejz0CWMwEX1cBFtiAVm+g7n5aHNG+dO5uJUW3GtpbxNIKn zhO8GKREp8h4CWmd2TFpM4ENVI6PJYCf26pUzcjvnfJLFuNU2xpTNgtltgVDjMeC/hYD 6+6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694881351; x=1695486151; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=a+yBHB7NxTpsynDoEBqNFk6YAWtINqaaQmccUbmFZv8=; b=hrx7X91W50jWdN/fNoLPUwcloXDcQYsN5a4s4F9r4lxsEnjoooQM2SliaO/efcNEEp +ZH+ZtW7QcupqLzGFaHY8JcPi2tr8xSba8Mz3Lg0m72kraG+MKBZArGA79r/dDaGmeiW 5St+39ZMHxpgGD1qc1N96aX2eSWPop9j7PVtfvPEdSOuP5Hs9ty7WaDwTuz8H6FD8Zxf GmrJ92PghdXkwEqGohq6Gm9CDCJHas5UoQ1qUF6PBtYfk4q/tgSCdZKiYjiA2z0V1wnk BqJpg++3P03y4gtgpyq7N1M6SMQlJKwjkiVO0Mu5WeS1UJIipqSHA8xJ+pKeNLyxQKmJ RhGQ== X-Gm-Message-State: AOJu0YzixiIROV2Yln34hPqR9N7Kjsn5FcUOX1wQjfGHQLD8ixGMcvI6 COBOymF6WtMQu5316WKejfEEaGCIzCQcbtMWiisW7Q== X-Google-Smtp-Source: AGHT+IFXG6kDtuHO0ywP9hnaSxY0gbcylQZ8lAkMCJgPh25xhL2boeIGj7bDB8U9v8j79aC0R0vdXoqM5jtaVlG7Spg= X-Received: by 2002:aa7:c05a:0:b0:52d:25df:e8c3 with SMTP id k26-20020aa7c05a000000b0052d25dfe8c3mr3895863edo.27.1694881351349; Sat, 16 Sep 2023 09:22:31 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <7EEF3435-064D-4C3C-98E4-2B27A788DB43.ref@yahoo.com> <7EEF3435-064D-4C3C-98E4-2B27A788DB43@yahoo.com> <4D51E8E6-8AF0-4773-A9BA-D53C08B744EA@me.com> In-Reply-To: <4D51E8E6-8AF0-4773-A9BA-D53C08B744EA@me.com> From: Warner Losh Date: Sat, 16 Sep 2023 17:22:20 +0100 Message-ID: Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature To: Toomas Soome Cc: Mark Millard , void , Current FreeBSD Content-Type: multipart/alternative; boundary="000000000000e24f0206057c50cf" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4RnxDC36lnz4n42 --000000000000e24f0206057c50cf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Sep 16, 2023 at 5:11=E2=80=AFPM Toomas Soome wrote: > > > > On 16. Sep 2023, at 18:43, Mark Millard wrote: > > > > void wrote on > > Date: Sat, 16 Sep 2023 12:12:02 UTC : > > > >> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote: > >> > >>> Yes. The boot loader comes from the host. It must know how to read > ZFS. > >> > >> It knows how to read zfs. > > > > I expect Warner was indicating: you have a (efi?) loader that knows > > how to deal with the features listed in: > > > > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd > > > > being active but not with some new feature(s) listed in: > > > > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > > > > being active. > > > > The following are the "read-only-compatibile no" features > > that are new in openzfs-2.2 compared to openzfs-2.1-freebsd : > > > > blake3 > > ednor > > head_errlog > > vdev_zaps_v2 > > > > So any of those being active leads to lack of even read-only > > activity being compatible. (Although, the loader's subset > > of the potential overall activity might allow ignoring some > > specific "read-only-compatibile no" status examples.) > > > > For reference: > > > > # diff -u99 > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-f= reebsd > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > > --- > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-f= reebsd > 2021-06-24 20:08:57.206621000 -0700 > > +++ > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > 2023-06-10 15:59:25.354999000 -0700 > > @@ -1,34 +1,40 @@ > > -# Features supported by OpenZFS 2.1 on FreeBSD > > +# Features supported by OpenZFS 2.2 on Linux and FreeBSD > > allocation_classes > > async_destroy > > +blake3 > > +block_cloning > > bookmark_v2 > > bookmark_written > > bookmarks > > device_rebuild > > device_removal > > draid > > +edonr > > embedded_data > > empty_bpobj > > enabled_txg > > encryption > > extensible_dataset > > filesystem_limits > > +head_errlog > > hole_birth > > large_blocks > > large_dnode > > livelist > > log_spacemap > > lz4_compress > > multi_vdev_crash_dump > > obsolete_counts > > project_quota > > redacted_datasets > > redaction_bookmarks > > resilver_defer > > sha512 > > skein > > spacemap_histogram > > spacemap_v2 > > userobj_accounting > > +vdev_zaps_v2 > > +zilsaxattr > > zpool_checkpoint > > zstd_compress > > > > (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does > > not exist yet. Thus were I had the diff look.) > > > >> On the host in question, there are many guests, > >> some with zfs-boot, some not, just file-based. > > > > But with what openzfs features active vs. not active > > in each case? > > > >> What the host is not, is zfs-on-root. It boots from ssd (ada0). > >> The vdevs are on a sas disk array. > >> > >>> So either your bootable partitions must not have > com.klarasystems:vdev_zaps_v2 > >>> in your BEs or you must have a new user boot. I think you can just > install > >>> the one from 14, but haven't tried it. > >> > >> Can you briefly explain how I'd install the one from 14 please? > > > > > > I do not use bhyve so I do not even know if the > > context is using the efi loader from a msdosfs > > vs. not. For efi loaders, copying from one msdosfs > > with a sufficient vintage to the one with the wrong > > vintage (replacing) is sufficient. > > bhyve in freebsd is traditionally using /boot/userboot.so, I believe. Yes. We use the *HOSTS* (running FreeBSD 13) /boot/userboot.so to boot the FreeBSD 14 image. Since we're not using the boot loader from the target image to load it for bhyve, the loader we're using has to understand the ZFS dataset that it's booting off of. FreeBSD 13's userboot.so doesn't support all the bells and whistles that the ZFS folks have added to 14. So, either you have to turn off those features (which I got no clue how to do in the normal installer), or you have to update userboot.so to the FreeBSD 14 version (which I think had a good chance of actually running on FreeBSD 13 since it has no 'system' references, which are confined to bhyveload). Warner > > > > # find /boot/efi/EFI/ -print > > /boot/efi/EFI/ > > /boot/efi/EFI/FREEBSD > > /boot/efi/EFI/FREEBSD/loader.efi > > /boot/efi/EFI/BOOT > > /boot/efi/EFI/BOOT/bootaa64.efi > > > > There may well be only: > > > > EFI/BOOT/bootaa64.efi > > > > for all I know. > > > > From an amd64 context: > > > > # find /boot/efi/EFI/ -print > > /boot/efi/EFI/ > > /boot/efi/EFI/FREEBSD > > /boot/efi/EFI/FREEBSD/loader.efi > > /boot/efi/EFI/BOOT > > /boot/efi/EFI/BOOT/bootx64.efi > > > > There may well be only: > > > > EFI/BOOT/bootx64.efi > > > > for all I know. > > > > (I set things up to have the EFI capitalization > > so that referencing efi/ vs. EFI/ in my context > > is unique for the mount point. vs. the msdosfs > > directory.) > > > > =3D=3D=3D > > Mark Millard > > marklmi at yahoo.com > > > > > > > --000000000000e24f0206057c50cf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sat, Sep 16, 2023 at 5:11=E2=80=AF= PM Toomas Soome <tsoome@me.com> = wrote:


> On 16. Sep 2023, at 18:43, Mark Millard <marklmi@yahoo.com> wrote:
>
> void <void_at_f-m.fm> wrote on
> Date: Sat, 16 Sep 2023 12:12:02 UTC :
>
>> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote:
>>
>>> Yes. The boot loader comes from the host. It must know how to = read ZFS.
>>
>> It knows how to read zfs.
>
> I expect Warner was indicating: you have a (efi?) loader that knows > how to deal with the features listed in:
>
> sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
>
> being active but not with some new feature(s) listed in:
>
> sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
>
> being active.
>
> The following are the "read-only-compatibile no" features > that are new in openzfs-2.2 compared to openzfs-2.1-freebsd :
>
> blake3
> ednor
> head_errlog
> vdev_zaps_v2
>
> So any of those being active leads to lack of even read-only
> activity being compatible. (Although, the loader's subset
> of the potential overall activity might allow ignoring some
> specific "read-only-compatibile no" status examples.)
>
> For reference:
>
> # diff -u99 /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.= d/openzfs-2.1-freebsd /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibi= lity.d/openzfs-2.2
> --- /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzf= s-2.1-freebsd 2021-06-24 20:08:57.206621000 -0700
> +++ /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzf= s-2.2 2023-06-10 15:59:25.354999000 -0700
> @@ -1,34 +1,40 @@
> -# Features supported by OpenZFS 2.1 on FreeBSD
> +# Features supported by OpenZFS 2.2 on Linux and FreeBSD
> allocation_classes
> async_destroy
> +blake3
> +block_cloning
> bookmark_v2
> bookmark_written
> bookmarks
> device_rebuild
> device_removal
> draid
> +edonr
> embedded_data
> empty_bpobj
> enabled_txg
> encryption
> extensible_dataset
> filesystem_limits
> +head_errlog
> hole_birth
> large_blocks
> large_dnode
> livelist
> log_spacemap
> lz4_compress
> multi_vdev_crash_dump
> obsolete_counts
> project_quota
> redacted_datasets
> redaction_bookmarks
> resilver_defer
> sha512
> skein
> spacemap_histogram
> spacemap_v2
> userobj_accounting
> +vdev_zaps_v2
> +zilsaxattr
> zpool_checkpoint
> zstd_compress
>
> (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does
> not exist yet. Thus were I had the diff look.)
>
>> On the host in question, there are many guests,
>> some with zfs-boot, some not, just file-based.
>
> But with what openzfs features active vs. not active
> in each case?
>
>> What the host is not, is zfs-on-root. It boots from ssd (ada0). >> The vdevs are on a sas disk array.
>>
>>> So either your bootable partitions must not have com.klarasyst= ems:vdev_zaps_v2
>>> in your BEs or you must have a new user boot. I think you can = just install
>>> the one from 14, but haven't tried it.
>>
>> Can you briefly explain how I'd install the one from 14 please= ?
>
>
> I do not use bhyve so I do not even know if the
> context is using the efi loader from a msdosfs
> vs. not. For efi loaders, copying from one msdosfs
> with a sufficient vintage to the one with the wrong
> vintage (replacing) is sufficient.

bhyve in freebsd is traditionally using /boot/userboot.so, I believe.

Yes. We use the *HOSTS* (running FreeBSD 13) /bo= ot/userboot.so to boot the FreeBSD 14
image. Since we're not = using the boot loader from the target image to load it for bhyve,
the loader we're using has to understand the ZFS dataset that it's= booting off of. FreeBSD
13's userboot.so doesn't support= all the bells and whistles that the ZFS folks have added
to 14.<= /div>

So, either you have to turn off those features (wh= ich I got no clue how to do in the=C2=A0
normal installer), or yo= u have to update userboot.so to the FreeBSD 14 version (which
I t= hink had a good chance of actually running on FreeBSD 13 since it has no &#= 39;system'
references, which are confined to bhyveload).

Warner
=C2=A0
>
> # find /boot/efi/EFI/ -print
> /boot/efi/EFI/
> /boot/efi/EFI/FREEBSD
> /boot/efi/EFI/FREEBSD/loader.efi
> /boot/efi/EFI/BOOT
> /boot/efi/EFI/BOOT/bootaa64.efi
>
> There may well be only:
>
> EFI/BOOT/bootaa64.efi
>
> for all I know.
>
> From an amd64 context:
>
> # find /boot/efi/EFI/ -print
> /boot/efi/EFI/
> /boot/efi/EFI/FREEBSD
> /boot/efi/EFI/FREEBSD/loader.efi
> /boot/efi/EFI/BOOT
> /boot/efi/EFI/BOOT/bootx64.efi
>
> There may well be only:
>
> EFI/BOOT/bootx64.efi
>
> for all I know.
>
> (I set things up to have the EFI capitalization
> so that referencing efi/ vs. EFI/ in my context
> is unique for the mount point. vs. the msdosfs
> directory.)
>
> =3D=3D=3D
> Mark Millard
> marklmi at yahoo.com
>
>


--000000000000e24f0206057c50cf--