From nobody Tue Sep 19 03:39:29 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RqS8f275dz4tGNQ for ; Tue, 19 Sep 2023 03:39:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-21.consmr.mail.gq1.yahoo.com (sonic313-21.consmr.mail.gq1.yahoo.com [98.137.65.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RqS8d6vkLz4NmW for ; Tue, 19 Sep 2023 03:39:45 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1695094783; bh=Oi5TT993gd4R5VhSYUQiiVqsdZz+5UfHitmofdYlnTk=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=jBQBlC57GLbiicxxWwFXQc2LHxSGa9bvUkxe3lhQCg/JDKtFbISvZPpCYujQ4sEUWJKec01AbPGkdfp8X8nzGybNBYPO0nGu57Ozb0p+2vAPFJbq3i1akdpf/JdQVHHMjC9dnrMDI3wKYufRX92CvfNcoO25Gwdzv6LmibnQR+uVdouKBuIAq4vIHxCAAm7YVJQf45UYnVwa6kCbgzNxtYfOG5Lsia6BvzEPDCBeUOXZtSfHKKtiAsKyrcMYUXtO3869XK7p1ex6jlvUJBl73RwJFM4GMzJG7/MV8banOVrLnhVjWFTHE+a/AIx85y4mrGnwfHrchs561sp2qonMnw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1695094783; bh=XhXifNV+lbYP2AibIeXVI265H2Go66Pxwa2ZzfCgrPg=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=uVt3cMpM1C5UCbgI91VCM8sUBPDidjmgZgRoYKGXJLktOpINGpJCYox5OY4sZaEbtc1jhV9KJW3wlKuw1X+lRgKoPg7iMt0a/kw1nQGoaAySHvKfBhs0GUiYwN9AESRWwODLoAMJvDqmOi0Hxm/iE7Ttjr6mN+RWxcPkAmoPutf/73VZFuqfyZ6Toqvdsljei0rh3uRL2oUN290BP+Id+akoZBuPnfmB9bTWE4JYvjjVIpO044Lv3+zTUdKq26vmMiqwX9Z+GRLt92I9pnFyjO0bi3fM7R4IK2C9iHdZSIqP6R7e/UKowhBKk/uw0rPs0jl+WiOAlXpH8pBgaytbEQ== X-YMail-OSG: vnJHSRkVM1k6gFtdfiYq1kM2hcu_c15gzVKeCmUJmAC6S3UxRLGbav5PRBl1JSG IaksYWVNt2YN.8b9jx2dRM7htZ5M48p6t3MfohjzT_O2T6nEXq0XOjGu5lG9KQrNOp7zRvPFLoOn HMM5n9o_vyGuZJ3j.K7KI6UOILkYhNc_y4MBLsUNoIvx53vLXlHD20v0tp.EdIHoT9D.t8kC0MW7 1T9B.yKl6hSurSggPbaIK9qC7YFRiI0eWvssvawBZF40fZr1I2tGcaEAc8Q7Rrdm_pRSX60hN.VK snyaYvgPtnIlK1p20y8J7ZqnR79yWccoYRsbMMZkomYPQQtbYo5qw4dMNHVAcLlMV50uOzoFYXU2 AY_ieXNpCo3t6C90kTJMHIuxWNKT5xI4YBB_QxnHp6y6NgreoALqdKHOngvLbJxySxH29HO6INCL IOeCDYPzeNUKcYjr02cIkHVJefHJk718PwyyvgnGEs882DoQ6Xaj9fRQEsLdaCZ__0a.HBDQvp28 xsnrvlHpcFJdTpcKp5CslJcIqJLWmLO0ttg_Q6Flcxn65E_WqmmUNNru3ZoHRMNG1DpRx.fcDUDU I21uRLjW8YCbNLwg22Vk9CiMMzl6FZPQb8ujUUKAVGjipOI8X0scqrW6iR8iYwuj09Ts01DOG_sX uA5q6quIsxDHaddH9FIT8nKUoz5fV1mzBYpOT3iON9qLrQ1PqDrDEuRk7x2OZV9bhcVVEpma1KYU l2Ahi.LRXII.LOeAIq08lQfcOAnvdU4ZSR.fd_vCFGS_pPTCIglhm3KO_RwV4pS92W2qu_xZxy1a 1eGpS_7EQw64n4HMcRiXY8kow17Yy2jL.rrSGJfR8hHrgFtOLjU7co838hI5umBcy2B6Ndt80fBH E9hRCfXAgDcE2qMh3ZvseYKcuh1Psl2cWONHjkrs3SNdxXg.txov12wS19kCD.4gaGEv5XtNAoNH fMXNOmus2EPxDAjTVTxPtuv2uGnerIXgwZWLxy6q5Zu43b9jpSossxG9oToAtx.4TwFjYr_KnM5B NCVVzitwHekbwxUH.EsgfMPnZYoD7kWuYXM_3pDOFXE3pbMb.D3DoPPGN8NQhM6RK6CbVK6fTAN2 h9f9qGDFMAObhlKIjXEZw3g4I.aFberoSGUPHqAYOBdz1iVJVV.fJkPKrcU6AvdI4CXmieV6Tc1V Q6fu2oOqzChJWjjWA1TS6sIdAlKkQOB8IdZlEojD4yVR08s7MghlcYdosbYimwdEUWUrggkbj7Tq AFZeCCF_N626DSSYK6zjAi_05Vo.ikjczlwGTBG1CilSTvJUlMLso07alBOgU_jorxOxeHbO0zIJ 9JnOzEmcQSZHky0ZkynWktjO76r5DOSx7BTmEu5yjW1HOY.Rk51maY9_pe2RlgYLMusNA.8sJSfY ZXiaQK5Z4.8UbCCbQ.4ASTX6zWN374ZIwk3GPa32fCnmPjhTlhZrLez9Cp_nxgUVhy2cvEjKitif V5moP.mDP1GmSa.C97ivpol.KcgLyNSOvdWIjs0H_1IbJM.MisiLA4sLQOVyN8h0hikBhoJ4EPEt fz1YcOCQiCgpvTnXWUcLXEiyfXCEjUzvyLm22miQSBvdUzy2oeGExabOm8SX5jtnFeKahtWh4F1w qq4FRH3sEWgFlhKuEt.L2VYsExe39xClxHgOrKyLjmsyXHnmZlxPp1GezXl1UH._ZWlZzDfbbtoj xTY5lDFA2_0LRPmbVhTAl1X0kn2WRI7EFGAvNW0U.wSxSTM9UpX551zDWo7PSYwFuovr8OT4Bayf TICGmeN_engFoiKdPYODNB.Xqv7jG.LgZ9AC8obqZOASgcZR3IPP76rY4N1jeLKFHyo44eb.eTiJ IsKtcVV8VcQDK6ZImHhShgyCQ551Gkd9NV_KhQwSDzfxClcH26_scZJJN_CZNaTjAA8qXtzNHY4V mfEAbjpfM_lJy.dHVKmfgSHeji7j4_etTIRKTlHaL.Kzr9wGhxYVm5T2uSl_SqtluDxNls9re1nJ kIppHhHT0kA1ZdfjZ6_Dcn_xp7e7EVMnof4azAQyJ20FFxqj3gzlrtSoDFcmBYEW0hQpXp1WpT_K 0fHSRrHoyHLUaecjJvQNcrjyk50kmQ4Czwg27uQr6DGi3.6MfpBacDpq8J1R1.Nr56pNh3OQSK7X I4VKc0rqlb2QEKwp4Q3u3T1DvDwP3cFqsVpUFrev6P9qInnA8Yydmx_f4x28lX.LW1_xJfBXxv8i nZ9rR9y_0E80C3riwuaVf_Uhb.UpWf.EoWlGvDfr2uUt.KkSthp4KtbRKOCYf2jHCdLyLyBza2B6 9GuA- X-Sonic-MF: X-Sonic-ID: f0bfc2fc-31c4-4f25-8e4e-c6099a7802ad Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Tue, 19 Sep 2023 03:39:43 +0000 Received: by hermes--production-gq1-77657878bb-wlcsw (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 9b015dac059db35e12762e947d76a153; Tue, 19 Sep 2023 03:39:40 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: vfs.zfs.bclone_enabled (was: FreeBSD 14.0-BETA2 Now Available) [block_cloning and zilsaxattr missing from loader's features_for_read] From: Mark Millard In-Reply-To: Date: Mon, 18 Sep 2023 20:39:29 -0700 Cc: Current FreeBSD , Warner Losh , Glen Barber Content-Transfer-Encoding: quoted-printable Message-Id: <12D5CD6A-34ED-47E2-B43E-37AD5582206C@yahoo.com> References: <5C769ACC-F264-4BAB-AF7B-8C463A4BD99E@yahoo.com> To: Alexander Motin X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Queue-Id: 4RqS8d6vkLz4NmW On Sep 18, 2023, at 17:05, Alexander Motin wrote: > On 18.09.2023 19:21, Mark Millard wrote: >> On Sep 18, 2023, at 15:51, Mark Millard wrote: >>> Alexander Motin wrote on >>> Date: Mon, 18 Sep 2023 13:26:56 UTC : >>>> block_cloning feature is marked as READONLY_COMPAT. It should not >>>> require any special handling from the boot code. >>>=20 >>> =46rom stand/libsa/zfs/zfsimpl.c but adding a comment about the >>> read-only compatibility status of each entry: >>>=20 >>> /* >>> * List of ZFS features supported for read >>> */ >> static const char *features_for_read[] =3D { >>> "com.datto:bookmark_v2", // READ-ONLY COMPATIBLE no >>> "com.datto:encryption", // READ-ONLY COMPATIBLE no >>> "com.datto:resilver_defer", // READ-ONLY COMPATIBLE yes >>> "com.delphix:bookmark_written", // READ-ONLY COMPATIBLE no >>> "com.delphix:device_removal", // READ-ONLY COMPATIBLE no >>> "com.delphix:embedded_data", // READ-ONLY COMPATIBLE no >>> "com.delphix:extensible_dataset", // READ-ONLY COMPATIBLE no >>> "com.delphix:head_errlog", // READ-ONLY COMPATIBLE no >>> "com.delphix:hole_birth", // READ-ONLY COMPATIBLE no >>> "com.delphix:obsolete_counts", // READ-ONLY COMPATIBLE yes >>> "com.delphix:spacemap_histogram", // READ-ONLY COMPATIBLE yes >>> "com.delphix:spacemap_v2", // READ-ONLY COMPATIBLE yes >>> "com.delphix:zpool_checkpoint", // READ-ONLY COMPATIBLE yes >>> "com.intel:allocation_classes", // READ-ONLY COMPATIBLE yes >>> "com.joyent:multi_vdev_crash_dump", // READ-ONLY COMPATIBLE = no >>> "com.klarasystems:vdev_zaps_v2", // READ-ONLY COMPATIBLE no >>> "org.freebsd:zstd_compress", // READ-ONLY COMPATIBLE no >>> "org.illumos:lz4_compress", // READ-ONLY COMPATIBLE no >>> "org.illumos:sha512", // READ-ONLY COMPATIBLE no >>> "org.illumos:skein", // READ-ONLY COMPATIBLE no >>> "org.open-zfs:large_blocks", // READ-ONLY COMPATIBLE no >>> "org.openzfs:blake3", // READ-ONLY COMPATIBLE no >>> "org.zfsonlinux:allocation_classes", // READ-ONLY COMPATIBLE = yes >>> "org.zfsonlinux:large_dnode", // READ-ONLY COMPATIBLE no >>> NULL >>> }; >>>=20 >>> So it appears that the design is that both "no" and "yes" ones >>> that are known to be supported are listed and anything else is >>> supposed to lead to rejection until explicitly added as >>> known-compatibile. >=20 > I don't think so. I think somebody by mistake added first featured = that should not be here, and then others continued this irrelevant = routine. My own development server/builder is happily running latest = main with ZFS root without any patches and with block cloning not only = enabled, but even active. So as I have told, it is not needed: >=20 > mav@srv:/root# zpool get all | grep clon > mavlab bcloneused 20.5M = - > mavlab bclonesaved 20.9M = - > mavlab bcloneratio 2.02x = - > mavlab feature@block_cloning active = local >=20 > Somebody should go through the list and clean in up from = read-compatible features and document it, unless there are some features = that were re-qualified at some point, I haven't checked if it could be. >=20 >>> This matches up with stand/libsa/zfs/zfsimpl.c 's: >>>=20 >>> static int >>> nvlist_check_features_for_read(nvlist_t *nvl) >>> { > ... >>> rc =3D nvlist_find(nvl, ZPOOL_CONFIG_FEATURES_FOR_READ, >>> DATA_TYPE_NVLIST, NULL, &features, NULL); >=20 > Take a note it reads ZPOOL_CONFIG_FEATURES_FOR_READ. Same time = features declared as READONLY_COMPAT are stored in FEATURES_FOR_WRITE, = that boot loader does not even care. >=20 >>> I do not know if vfs.zfs.bclone_enabled=3D0 leads the loader >>> to see vs. not-see a "com.fudosecurity:block_cloning". >=20 > bclone_enabled=3D0 block copy_file_range() usage, that should keep the = feature enabled, but not active. It could be related if the feature = would be in FEATURES_FOR_WRITE, but here and now it is not. >=20 >> It appears that 2 additions afeter opebzfas-2.1-freebsd are >> missing from the above list: >> com.fudosecurity:block_cloning >> org.openzfs:zilsaxattr >=20 > Nothing of ZIL is required for read-only import. So no, it is also = not needed. Thanks for the details in your notes, including that bclone_enabled=3D0 is not relevant. As a result I found out about zhack feature stat POOLNAME that shows the ZPOOL_CONFIG_FEATURES_FOR_READ separately from then ones for write. Looking at the pool that I used for testing block_cloning via poudriere bulk build activity (and sorting the for_*_obj lists produced) . . . The for_read_obj list (with matching line added as a suffix): com.datto:bookmark_v2 =3D 0 "com.datto:bookmark_v2", = // READ-ONLY COMPATIBLE no com.datto:encryption =3D 0 "com.datto:encryption", = // READ-ONLY COMPATIBLE no com.delphix:bookmark_written =3D 0 = "com.delphix:bookmark_written", // READ-ONLY COMPATIBLE no com.delphix:device_removal =3D 0 = "com.delphix:device_removal", // READ-ONLY COMPATIBLE no com.delphix:embedded_data =3D 1 = "com.delphix:embedded_data", // READ-ONLY COMPATIBLE no com.delphix:extensible_dataset =3D 87 = "com.delphix:extensible_dataset", // READ-ONLY COMPATIBLE no com.delphix:head_errlog =3D 1 = "com.delphix:head_errlog", // READ-ONLY COMPATIBLE no com.delphix:hole_birth =3D 1 = "com.delphix:hole_birth", // READ-ONLY COMPATIBLE no com.delphix:redacted_datasets =3D 0 com.delphix:redaction_bookmarks =3D 0 com.joyent:multi_vdev_crash_dump =3D 0 = "com.joyent:multi_vdev_crash_dump", // READ-ONLY COMPATIBLE no com.klarasystems:vdev_zaps_v2 =3D 1 = "com.klarasystems:vdev_zaps_v2", // READ-ONLY COMPATIBLE no org.freebsd:zstd_compress =3D 0 = "org.freebsd:zstd_compress", // READ-ONLY COMPATIBLE no org.illumos:edonr =3D 0 org.illumos:lz4_compress =3D 1 = "org.illumos:lz4_compress", // READ-ONLY COMPATIBLE no org.illumos:sha512 =3D 0 = "org.illumos:sha512", // READ-ONLY COMPATIBLE no org.illumos:skein =3D 0 "org.illumos:skein", // = READ-ONLY COMPATIBLE no org.open-zfs:large_blocks =3D 0 = "org.open-zfs:large_blocks", // READ-ONLY COMPATIBLE no org.openzfs:blake3 =3D 0 = "org.openzfs:blake3", // READ-ONLY COMPATIBLE no org.openzfs:draid =3D 0 org.zfsonlinux:large_dnode =3D 0 = "org.zfsonlinux:large_dnode", // READ-ONLY COMPATIBLE no I'll note that the following of the no-match examples are listed in openzfs-2.2 : redacted_datasets redaction_bookmarks edonr draid In fact, only edonr is new in openzfs-2.2 compared to openzfs-2.1-freebsd . So it appears that the 4 are missing from at least features_for_read but might be missing what is required to have them supported if something else is also needed. (The "=3D 0" might explain the lack of a complaint from the loader? Or is there more classification that I do not understand?) The for_write_obj list: com.datto:resilver_defer =3D 0 = "com.datto:resilver_defer", // READ-ONLY COMPATIBLE yes com.delphix:async_destroy =3D 0 com.delphix:bookmarks =3D 0 com.delphix:empty_bpobj =3D 44 com.delphix:enabled_txg =3D 34 com.delphix:livelist =3D 0 com.delphix:log_spacemap =3D 1 com.delphix:obsolete_counts =3D 0 = "com.delphix:obsolete_counts", // READ-ONLY COMPATIBLE yes com.delphix:spacemap_histogram =3D 110 = "com.delphix:spacemap_histogram", // READ-ONLY COMPATIBLE yes com.delphix:spacemap_v2 =3D 1 = "com.delphix:spacemap_v2", // READ-ONLY COMPATIBLE yes com.delphix:zpool_checkpoint =3D 0 = "com.delphix:zpool_checkpoint", // READ-ONLY COMPATIBLE yes com.fudosecurity:block_cloning =3D 0 com.joyent:filesystem_limits =3D 0 org.openzfs:device_rebuild =3D 0 org.openzfs:zilsaxattr =3D 3 org.zfsonlinux:allocation_classes =3D 0 org.zfsonlinux:project_quota =3D 44 org.zfsonlinux:userobj_accounting =3D 44 So: All those "READ-ONLY COMPATIBLE yes" examples do not need to be listed in features_for_read . The following have no matches in either for_*_obj list that zhack produced: "com.intel:allocation_classes", // READ-ONLY COMPATIBLE yes "org.zfsonlinux:allocation_classes", // READ-ONLY COMPATIBLE yes I'll note that openzfs-2.2 and openzfs-2.1-freebsd both list: allocation_classes But the "READ-ONLY COMPATIBLE yes" status suggests this is not a loader functional problem but may be a waste: all the "READ-ONLY COMPATIBLE yes" could be removed from features_for_read . Thanks again. =3D=3D=3D Mark Millard marklmi at yahoo.com