From nobody Mon Sep 04 09:56:53 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RfPF42957z4rRwr for ; Mon, 4 Sep 2023 09:57:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-23.consmr.mail.gq1.yahoo.com (sonic303-23.consmr.mail.gq1.yahoo.com [98.137.64.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RfPF23Qhnz3Cqw for ; Mon, 4 Sep 2023 09:57:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=iHWcXbE1; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.204 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693821428; bh=NFiWHdIsJzWn1ErCVl7kd+aBOElqxl6VmMbadABmM2A=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=iHWcXbE1IYKP3w9qHrOluJ2V8DIPPoxmOecipf4ePMzqh29oe2ukQtJlQkLt9AgMZMPKq5oautSZC+vBN+EiUt+89uTB6wE78rokkB2LCPfLF0IeiY1Zc+5HJ52FrlcLGnXEzTd4Fg7UAqY5Q+SrWp/FQcNa0VaVgS22tkVz/Egp4+2Gkufs4No3RqpT+Cb50dN8e9GzheS4XOPypOQOt/SuOsQchjOfVAWSnkswpHQWVcr0fBlnJNLoTO+srmSA6082OfuKooyMTQsewu5rbW2lh5Mdvk2ZuQhb+vSxCICZy6HBeBs5D60zg71tiU4eLPos+baQYn6lXH9BX2UjfA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693821428; bh=CJUNwyW7sprvM9gODyQOUN09iHl+6NOb8BYGg3YnCMz=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=RvFV9lYXdDqaw2HaIZAgHCaVXaUF2fcDYcXaZGNMYxdOOaUa1XfZiXWRy37rqgFrxlQGd/ZUdKy5C3kuMoBdntIGsufYffvWi/oEEgPxYpXVy0nban3pXY89fum3Pjt/hNSyrrghNU7+eoaiBCPDADcoPtua5SkEu96D8fFKlG4vWYwQBbGscMT4sncdMolEM+t53gdDskj9ytR8VQMvlGMagkM9j0pZXyo9KbPPshluXptK1haSHYiSAHegNXb1/IMf1/bPrYv6H49BYR3jhRNPmn8hniKSut/D9b4P4rJS13ispCW3uMUNBBRms+Z6cJcjjZXi0J7LQ6Nn3t5OUQ== X-YMail-OSG: uvJb7sMVM1kJ1TvOKlB5RVRP9iq0mP97tuWMBlnnQWZtFcCsKp.eCDr6fsV6otc V0hlc5GonkH.p1EnKA8zGKLCaPtMMJ4c1OHziuHv9R53SNFgb2BskSYduDRK6Of8NBnGxkbaAl.W K.WGsfMsT_vGE1I1_qftf8jXYms_Yba3C7k_kmaFIEaLYcIshq3fQcrbt_HRwfYZ39D1OOjXJ87N DIP15rehie3Qy.oInzGVNYwxTr_iVpMAZomk1rEmwZu3Jub7vR2v235d09QTpm7SFHQDT51hIpVT 8d2FTB2lh0r6OAjGiZd9ubG5On4LOcf0FgE7qHikqkPYKHHZdbxixRtcpLMrjPtQJU4FScad0DE7 eVe1rKfl002AzlteQwa3FReENIA0.mDkobjcjnnwFKOvHOi9Wvy5eWkeRF6.IH2gIaZKg3jt18ii IUFKNv9gSXKXqPrYtB9hZ2qTauY2n5TfPw5KgqKxk60HvRqVHBfqLohk4hLrz3WruofMObOMF5kf na.NyY3CTPAq.Ah6Jg8LIO38Wjy9xSlFtQiS6pxESuRE3QphOyJZQuUUR2g2HqPRhC6hY8BY.eu9 ZAWS3Pb3RX6yC7cqXSh__Zf4jFch6ABl_aiNHyjnnVNc9NvSStVAyYnUInUDxyn4nr5Rr_kpaSWe CvmKUlAjM4NBRizlwJgSjP085uhNHGEndgXG93lSINVqoZkXY2GMWZAu3P1t.BHcsuGSL7eMCf7Y mw050NgSx.0mVQUKYMdu5_eWQmQfEKTrzHB9uoByOPIOhHoCNu3Ru0caP97vCDYmQmDyDGLi3GpO KBE7xUzhgI6ZmDYoS0NM.3L4XmwAopR2U0sHdeOuTmDJLE8d.ZT8QldLVG0PO5VH.9mVDNIeLJWA f_ublXf851eHrVv6POInhx5Mw.tvBSj4k1kek_hFQ857bjEqEj9PemmQcDXmhOPbL9PSGZoVditM yzD03j4d0am0iJS.WqI7lb6NCpZuhDSFkwWyU2s0N96EsPdP7ptvYUzfRcunmAR5txtwdnmbsnKY M8ePcvlP5GU2mPicVJdkIVKMMocWnfOkfTI6PyY0JKlbLTUTt3ty4bpSS3F.5_uNVi_e7sH7H6SU EeZXVLVLT4DO5iCgwmOOPkkw7qODSvQH2ItI3vF7A9_Qt.UBtELjf5aNW_ECjEfd0v0uYYPNaVBa UIFFlHYf9H0FqBifEUeYBUwzHJgbNuSTnP59PzMSsuIbGG0JIcyqTxivY3nNg4Nh6nVGl9l2HL84 76T3GV3qx3EwHvG2Z5tBtjwTHWG7PsMWIeVszPgglfAmWH7RJEDsKx.HIJglY5YcGUpBS31_j1In iBFA7uH.qrAcDlI43ozjzxym1eLgG.ncd24uF_GgsKmMHDTG7IHE_BapSgcpmgc7gohH2aJF7c81 grMYFw7CWpr5dL9k9PhkyCg6V3JP2GrQA0nrfU5Xo_1Jqqytposg3cR15XIO5mBfDcKgp9MExRIu gJ5DWh2MdKYLXbgdqIq3oiU0X1plai17RnFonIMS4T.xw_c3q7_4N9UVFV912GnNkHOS33VNRZ9c 50pfaIwK3_t6PKc6yEdS2rucdb8r2VXZ9Qe9dQEZCbWl339FOOakeYrJ8ndEsuqZusItF7gFS3lY yOcg4qh9rzY9ttG46KByzLr6pB2APsEd52_y4uonNLfF1EG14HrVLeSPeFwRwmnbKPEjC2QsO5gK h.dD5n1FSaaU_hmI8Y4tuy0T.wPT4UjPw7Lx1moNu6_LN3DD1QG7EABiTq.HOtzXO0yUkk6BSHx2 otRG6x9MidALbbH0yro.ZSn4Dsklo9CBx5uK4XLHqJ9_qBOwo95wwFadim.oDqa3dY8JVgKAAQSa h5IvgJUdk2kENZJ9B8GEK0iHNq0QPPZLOHZnvbFpAwIziyKtjDstjYaVjt349vP0JEAUQXmzylzd O_RKnlmV.qko7WageZ3WNFstisXg4g8YfUi3SgBnaLjItI_1XATbxVZvPXqPU8TR5a89c013ws.y aYTSe0my8YZGMU9p14fWQ.oECju748C1lxDE3ThY.IxoASy44nIe1kDdVN2TqSNjPWEO39.UfZ_H JQ1luF7gNcBhgKts786fJwjevbczJh68nU.alQhvlwy4ZG.JBTOrr99Qis986FnJsDSIUotiK7XB ybyg.QQRDSDepqCCnynmg7asfsFqM9okaidpHlVVEgcc2cfVx_aYJXJRmgIsmmBlYyKynn_bcXRV 6oxmdHJngbQY6kFtpZYN3m9_HEuJvWgxw_oVzNm5qJsNAcGtyB74mJR8aSQhFK.ZBiCaDhHhJxTk TVlw- X-Sonic-MF: X-Sonic-ID: 564e4d60-190f-4343-8ab5-678944ecda5e Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Mon, 4 Sep 2023 09:57:08 +0000 Received: by hermes--production-gq1-6b7c87dcf5-rj56s (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID bd7e704fa70995972c77a0f07d2f2e73; Mon, 04 Sep 2023 09:57:03 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: An attempted test of main's "git: 2ad756a6bbb3" "merge openzfs/zfs@95f71c019" that did not go as planned From: Mark Millard In-Reply-To: <2BDD30B5-6248-4EC3-83C8-0499E0717D1D@yahoo.com> Date: Mon, 4 Sep 2023 02:56:53 -0700 Cc: dev-commits-src-main@freebsd.org, Current FreeBSD Content-Transfer-Encoding: quoted-printable Message-Id: References: <673A446E-6F94-451E-910F-079F678C5289@yahoo.com> <2BDD30B5-6248-4EC3-83C8-0499E0717D1D@yahoo.com> To: Alexander Motin X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.983]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; BLOCKLISTDE_FAIL(0.00)[98.137.64.204:server fail]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.204:from]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.204:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RfPF23Qhnz3Cqw On Sep 4, 2023, at 02:00, Mark Millard wrote: > On Sep 3, 2023, at 23:35, Mark Millard wrote: >=20 >> On Sep 3, 2023, at 22:06, Alexander Motin wrote: >>=20 >>>=20 >>> On 03.09.2023 22:54, Mark Millard wrote: >>>> After that ^t produced the likes of: >>>> load: 6.39 cmd: sh 4849 [tx->tx_quiesce_done_cv] 10047.33r 0.51u = 121.32s 1% 13004k >>>=20 >>> So the full state is not "tx->tx", but is actually a = "tx->tx_quiesce_done_cv", which means the thread is waiting for new = transaction to be opened, which means some previous to be quiesced and = then synced. >>>=20 >>>> #0 0xffffffff80b6f103 at mi_switch+0x173 >>>> #1 0xffffffff80bc0f24 at sleepq_switch+0x104 >>>> #2 0xffffffff80aec4c5 at _cv_wait+0x165 >>>> #3 0xffffffff82aba365 at txg_wait_open+0xf5 >>>> #4 0xffffffff82a11b81 at dmu_free_long_range+0x151 >>>=20 >>> Here it seems like transaction commit is waited due to large amount = of delete operations, which ZFS tries to spread between separate TXGs. >>=20 >> That fit the context: cleaning out /usr/local/poudriere/data/.m/ >>=20 >>> You should probably see some large and growing number in sysctl = kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay . >>=20 >> After the reboot I started a -J64 example. It has avoided the >> early "witness exhausted". Again I ^C'd after about an hours >> after the 2nd builder had started. So: again cleaning out >> /usr/local/poudriere/data/.m/ Only seconds between: >>=20 >> # sysctl kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay >> kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay: 276042 >>=20 >> # sysctl kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay >> kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay: 276427 >>=20 >> # sysctl kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay >> kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay: 277323 >>=20 >> # sysctl kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay >> kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay: 278027 >>=20 >> I have found a measure of progress: zfs list's USED >> for /usr/local/poudriere/data/.m is decreasing. So >> ztop's d/s was a good classification: deletes. >>=20 >>>> #5 0xffffffff829a87d2 at zfs_rmnode+0x72 >>>> #6 0xffffffff829b658d at zfs_freebsd_reclaim+0x3d >>>> #7 0xffffffff8113a495 at VOP_RECLAIM_APV+0x35 >>>> #8 0xffffffff80c5a7d9 at vgonel+0x3a9 >>>> #9 0xffffffff80c5af7f at vrecycle+0x3f >>>> #10 0xffffffff829b643e at zfs_freebsd_inactive+0x4e >>>> #11 0xffffffff80c598cf at vinactivef+0xbf >>>> #12 0xffffffff80c590da at vput_final+0x2aa >>>> #13 0xffffffff80c68886 at kern_funlinkat+0x2f6 >>>> #14 0xffffffff80c68588 at sys_unlink+0x28 >>>> #15 0xffffffff8106323f at amd64_syscall+0x14f >>>> #16 0xffffffff8103512b at fast_syscall_common+0xf8 >>>=20 >>> What we don't see here is what quiesce and sync threads of the pool = are actually doing. Sync thread has plenty of different jobs, including = async write, async destroy, scrub and others, that all may delay each = other. >>>=20 >>> Before you rebooted the system, depending how alive it is, could you = save a number of outputs of `procstat -akk`, or at least specifically = `procstat -akk | grep txg_thread_enter` if the full is hard? Or somehow = else observe what they are doing. >>=20 >> # procstat -akk > ~/mmjnk00.txt >> # procstat -akk > ~/mmjnk01.txt >> # procstat -akk > ~/mmjnk02.txt >> # procstat -akk > ~/mmjnk03.txt >> # procstat -akk > ~/mmjnk04.txt >> # procstat -akk > ~/mmjnk05.txt >> # grep txg_thread_enter ~/mmjnk0[0-5].txt >> /usr/home/root/mmjnk00.txt: 6 100881 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 _cv_wait+0x165 = txg_thread_wait+0xeb txg_quiesce_thread+0x144 fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk00.txt: 6 100882 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 = sleepq_timedwait+0x4b _cv_timedwait_sbt+0x188 zio_wait+0x3c9 = dsl_pool_sync+0x139 spa_sync+0xc68 txg_sync_thread+0x2eb fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk01.txt: 6 100881 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 _cv_wait+0x165 = txg_thread_wait+0xeb txg_quiesce_thread+0x144 fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk01.txt: 6 100882 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 = sleepq_timedwait+0x4b _cv_timedwait_sbt+0x188 zio_wait+0x3c9 = dsl_pool_sync+0x139 spa_sync+0xc68 txg_sync_thread+0x2eb fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk02.txt: 6 100881 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 _cv_wait+0x165 = txg_thread_wait+0xeb txg_quiesce_thread+0x144 fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk02.txt: 6 100882 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 = sleepq_timedwait+0x4b _cv_timedwait_sbt+0x188 zio_wait+0x3c9 = dsl_pool_sync+0x139 spa_sync+0xc68 txg_sync_thread+0x2eb fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk03.txt: 6 100881 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 _cv_wait+0x165 = txg_thread_wait+0xeb txg_quiesce_thread+0x144 fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk03.txt: 6 100882 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 = sleepq_timedwait+0x4b _cv_timedwait_sbt+0x188 zio_wait+0x3c9 = dsl_pool_sync+0x139 spa_sync+0xc68 txg_sync_thread+0x2eb fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk04.txt: 6 100881 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 _cv_wait+0x165 = txg_thread_wait+0xeb txg_quiesce_thread+0x144 fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk04.txt: 6 100882 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 = sleepq_timedwait+0x4b _cv_timedwait_sbt+0x188 zio_wait+0x3c9 = dsl_pool_sync+0x139 spa_sync+0xc68 txg_sync_thread+0x2eb fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk05.txt: 6 100881 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 _cv_wait+0x165 = txg_thread_wait+0xeb txg_quiesce_thread+0x144 fork_exit+0x82 = fork_trampoline+0xe=20 >> /usr/home/root/mmjnk05.txt: 6 100882 zfskern = txg_thread_enter mi_switch+0x173 sleepq_switch+0x104 = sleepq_timedwait+0x4b _cv_timedwait_sbt+0x188 zio_wait+0x3c9 = dsl_pool_sync+0x139 spa_sync+0xc68 txg_sync_thread+0x2eb fork_exit+0x82 = fork_trampoline+0xe=20 >>=20 >> (Hopefully that will be a sufficiently useful start.) >>=20 >>> `zpool status`, `zpool get all` and `sysctl -a` would also not harm. >>=20 >> It is a very simple zpool configuration: one partition. >> I only use it for bectl BE reasons, not the general >> range of reasons for using zfs. I created the media with >> my normal content, then checkpointed before doing the >> git fetch to start to set up the experiment. >>=20 >> # zpool status >> pool: zamd64 >> state: ONLINE >> status: Some supported and requested features are not enabled on the = pool. >> The pool can still be used, but some features are unavailable. >> action: Enable all features using 'zpool upgrade'. Once this is done, >> the pool may no longer be accessible by software that does not = support >> the features. See zpool-features(7) for details. >> checkpoint: created Sun Sep 3 11:46:54 2023, consumes 2.17M >> config: >>=20 >> NAME STATE READ WRITE CKSUM >> zamd64 ONLINE 0 0 0 >> gpt/amd64zfs ONLINE 0 0 0 >>=20 >> errors: No known data errors >>=20 >> There was also a snapshot in place before I did the >> checkpoint operation. >>=20 >> I deliberately did not use my typical openzfs-2.1-freebsd=20 >> for compatibility but used defaults when creating the pool: >>=20 >> # zpool get all >> NAME PROPERTY VALUE = SOURCE >> zamd64 size 872G = - >> zamd64 capacity 21% = - >> zamd64 altroot - = default >> zamd64 health ONLINE = - >> zamd64 guid 4817074778276814820 = - >> zamd64 version - = default >> zamd64 bootfs zamd64/ROOT/main-amd64 = local >> zamd64 delegation on = default >> zamd64 autoreplace off = default >> zamd64 cachefile - = default >> zamd64 failmode wait = default >> zamd64 listsnapshots off = default >> zamd64 autoexpand off = default >> zamd64 dedupratio 1.00x = - >> zamd64 free 688G = - >> zamd64 allocated 184G = - >> zamd64 readonly off = - >> zamd64 ashift 0 = default >> zamd64 comment - = default >> zamd64 expandsize - = - >> zamd64 freeing 0 = - >> zamd64 fragmentation 17% = - >> zamd64 leaked 0 = - >> zamd64 multihost off = default >> zamd64 checkpoint 2.17M = - >> zamd64 load_guid 17719601284614388220 = - >> zamd64 autotrim off = default >> zamd64 compatibility off = default >> zamd64 bcloneused 0 = - >> zamd64 bclonesaved 0 = - >> zamd64 bcloneratio 1.00x = - >> zamd64 feature@async_destroy enabled = local >> zamd64 feature@empty_bpobj active = local >> zamd64 feature@lz4_compress active = local >> zamd64 feature@multi_vdev_crash_dump enabled = local >> zamd64 feature@spacemap_histogram active = local >> zamd64 feature@enabled_txg active = local >> zamd64 feature@hole_birth active = local >> zamd64 feature@extensible_dataset active = local >> zamd64 feature@embedded_data active = local >> zamd64 feature@bookmarks enabled = local >> zamd64 feature@filesystem_limits enabled = local >> zamd64 feature@large_blocks enabled = local >> zamd64 feature@large_dnode enabled = local >> zamd64 feature@sha512 enabled = local >> zamd64 feature@skein enabled = local >> zamd64 feature@edonr enabled = local >> zamd64 feature@userobj_accounting active = local >> zamd64 feature@encryption enabled = local >> zamd64 feature@project_quota active = local >> zamd64 feature@device_removal enabled = local >> zamd64 feature@obsolete_counts enabled = local >> zamd64 feature@zpool_checkpoint active = local >> zamd64 feature@spacemap_v2 active = local >> zamd64 feature@allocation_classes enabled = local >> zamd64 feature@resilver_defer enabled = local >> zamd64 feature@bookmark_v2 enabled = local >> zamd64 feature@redaction_bookmarks enabled = local >> zamd64 feature@redacted_datasets enabled = local >> zamd64 feature@bookmark_written enabled = local >> zamd64 feature@log_spacemap active = local >> zamd64 feature@livelist enabled = local >> zamd64 feature@device_rebuild enabled = local >> zamd64 feature@zstd_compress enabled = local >> zamd64 feature@draid enabled = local >> zamd64 feature@zilsaxattr active = local >> zamd64 feature@head_errlog active = local >> zamd64 feature@blake3 enabled = local >> zamd64 feature@block_cloning enabled = local >> zamd64 feature@vdev_zaps_v2 active = local >> zamd64 feature@redaction_list_spill disabled = local >>=20 >> /etc/sysctl.conf does have: >>=20 >> vfs.zfs.min_auto_ashift=3D12 >> vfs.zfs.per_txg_dirty_frees_percent=3D5 >>=20 >> The vfs.zfs.per_txg_dirty_frees_percent is from prior >> Mateusz Guzik help, where after testing the change I >> reported: >>=20 >> Result summary: Seems to have avoided the sustained periods >> of low load average activity. Much better for the context. >>=20 >> But it was for a different machine (aarch64, 8 cores). But >> it was for poudriere bulk use. >>=20 >> Turns out the default of 30 was causing sort of like >> what is seen here: I could have presented some of the >> information via the small load average figures here. >>=20 >> (Note: 5 is the old default, 30 is newer. Other contexts >> have other problems with 5: no single right setting and >> no automated configuration.) >>=20 >> Other than those 2 items, zfs is untuned (defaults). >>=20 >> sysctl -a is a lot more output (864930 Bytes) so I'll skip >> it for now. >>=20 >>> PS: I may be wrong, but USB in "USB3 NVMe SSD storage" makes me = shiver. Make sure there is no storage problems, like some huge delays, = timeouts, etc, that can be seen, for example, as busy percents regularly = spiking far above 100% in your `gstat -spod`. >>>=20 >>=20 >> The "gstat -spod" output showed (and shows): around 0.8ms/w to 3ms/w, >> mostly at the lower end of the range. < 98%busy, no spikes to > 100%. >> It is a previously unused Samsung PSSD T7 Touch. >=20 > A little more context here: that is for the "kB" figures seen > during the cleanup/delete activity. During port builds into > packages larger "kB" figures are seen and the ms/w figures > will tend to be larger as well. The larger sizes can also > lead to reaching somewhat above 100 %busy some of the time. >=20 > I'll also note that I've ended up doing a lot more write > activity exploring than I'd expected. >=20 >> I was not prepared to replace the content of a PCIe slot's media >> or M.2 connection's media for the temporary purpose. No spare >> supply for those so no simple swapping for those. >=20 >=20 Trying -J36 (so: 32+4) got to 470 built in about an hour after [02] reached "Builder started". /usr/local/poudriere/data/.m used a little under 40 GiBytes at that point. (I do not have a file count.) The cleanup seems to have gone somewhat faster after my ^C for this context: ^C[01:20:20] Error: Signal SIGINT caught, cleaning up and exiting [01:20:20] [27] [00:02:54] Finished math/p5-Data-Float | = p5-Data-Float-0.013: Success [main-amd64-bulk_a-default] [2023-09-04_00h30m42s] [sigint:] Queued: = 34588 Built: 502 Failed: 1 Skipped: 50 Ignored: 335 Fetched: = 0 Tobuild: 33700 Time: 01:20:12 [01:20:22] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-09-04_0= 0h30m42s [01:20:23] [25] [00:04:46] Finished www/p5-HTML-TreeBuilder-XPath | = p5-HTML-TreeBuilder-XPath-0.14_1: Success [01:20:24] Cleaning up [02:17:01] Unmounting file systems Exiting with status 1 So it took about an hour to cleanup after 502 port builds into packages (not published, though). ( gstat -spod showed a fairly general, sustained lack of read activity, instead of the comparitively small sustained amount I'd not mentioned for the previous explorations. May be that helped. ) =3D=3D=3D Mark Millard marklmi at yahoo.com