From nobody Mon Sep 04 02:54:05 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RfCsG3jtYz4s33d for ; Mon, 4 Sep 2023 02:54:26 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RfCsF2xYwz4Yv7 for ; Mon, 4 Sep 2023 02:54:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=PkW5Tkud; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693796063; bh=FBi+sagkiLrDYCRxnDMG7OVrvlPWEK3CGsS4k1wrEAg=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=PkW5TkudUBMeFcl9gx90AYOW3BEkjxxt2ykXcse6LzA+GSyVi1GSrPQRSMNIb21ZE1s62Ry/bhQR1bqfudgsaa/Gn5Mqlq7brBKs2OoqTr1l6fVZ5YlDMWNcyGWRIxKiQeI7R2c1lOXWYfPvMzJ2zCC5+Zg2crNDPfcy/c2/jQpK0PeXm1SGijlmxl/g7ykY8kggEeIk2ymHguKRaiDA7JUNfgiDT/4N8YWRolnEO4wFS2ZlR92Ge9eLP3wJHsTs8coEQ0xqPtwTzt0d+ssxPsvxwsL6BQZRGgltu6gvBhxdcW3csQhfGITUHith2Q5PViF9Ct269FR0UKB9Yn3i/A== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693796063; bh=snKPMo7KCKLKHW0oxz7iZ/kXy9mpzaJgfBuLyAKPNq1=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=XWWRIJ6rDHPIQIXeWVts6P3/a0LdlTQrOanbO2cpp4/zbCv90DWDGYut5K3ZeJXnf4L7wSRxqEkL+Zn5eIu6Ek+BJtvZ1q4QkbX9zPkO0ygfpWWQh3BJib/tt3Mjp/4dJvCqUc8JJHinwUVeCs8Bol0/j3BL37u98BOQC8eeVNjhpqcGjcr0gSo0vy91asUBv4p/JUy9G62NZSFt2dSg8gk2KYoZ2tQXfq3IXM3jS1y4LWMIu1vTeluU9wgyYh8vp0qfaos54ccMOZSEfCLTRuCswN8p5/HRxk4Upmos0Gia5ArgjwxDJEseYal0lLUTR83DxvW9RLCynzDhMebg4w== X-YMail-OSG: RG38Y2kVM1n4vwBAirdHwtn_BtAWomD5bBVn5dLNHcgjm58Pc7ixDOPSgzpedaB pKDvjtGgtczssn0C2SaPCNk_O.oMC6ELquzVY8U9_.sk2nUORZ7aIEXUckr.ney5yJgzjoelqmtR HiSY4XH_GStmsoA3nENFtvI5E4A.7XkYK12ynh7cmhqcp5cJ4KDJnqaNqbOFZebO0f5f6Lo4JTIO clqzd.uicGOBOJeHGlmLEKIBIg4OKetx91YUOfkp.St7Ja5EHvDc0lJlG5JSFzBQ5S.S73nzD.Xu iWifH0RPOUamt9QnFyNRTbZ_bKrcYheXmoc4homdlqlKvOker8y7u9PodUuls7OdUuIdhjQIb.Sw hkPi1n.zc4MrDGaqmwC9IDhKqz4_tseerC8I1R78YzpHj70ngAQzJAXu15yxv6z.P1dqQn42wWlX UdqtiuIHKyi1QWA4epq8xk.AuvqvIDSVN2lngxw7OL.NQLxD9L.ZEzmJbFU04GeU93DZwaXNKX3z nmRTNwB7PZUveuNuUmDT2Jf9kqQMthln0VlijyuXgiAuwGthTV.ykOon9VvtVDVvN8sdgpCIACgb 8CCMkDUY2kt7.0IWBNqD3HSgrVk9NHwH0gV9A6pPNcy1QBE4AaKxTsvCtrBXqVxTzkJhnGMiieU7 nU8QXA.eYaaGRa0GewivQWsdC.NCdxYhteREXkPlJjfI5Z9usfSm3z.OvhbYVpfHv6AlIyaJmJYd YYfABw5ZT433u4Sr7i0ZvVYRnr09sNJbI1p3xGcZWfZ3HVHEqUoPm0.og9hie.Z4urAVdEGntH9C fE5nnj9RXV4vbN.fzqrdZ_ccdY3P3NTOj2XWDmXvA5.ZB1jb9yyzFW9YEKuna.MnQk54qijQaz9k AFhtLbuuQqIHoZfpf7q7_GZXbDeV440o2Rs_yVLdYlKlA2yv6PKjIxzMxgA.EpCgX507oMAAKs3J .IpNNosoC5wIeht_RKriGS3mrNHgz_648LvLCw0DvrpHehLOmV_GuPO7xs1Mhg1qp6LKACXkj0gK pHtp9Pce5Pn7EvTVg05fB9Uqembr006gA7Us_dpC90zKHEuI18.auTZFD1HqV_R5f7NOR8jRMqZD HhywoPfnfggOOxLa_o0B510OOUFClk5S7SWiWxItwn8O72dfGMlBFLRd8aTtz2JRQjO42m7EGoqS XT8v1kNq2nSS0dvUwxdP9DIapb4YjrcsqMlfVGiJrw_Gdb2z4_Gh9xviW7p.8ymvWPzusUskekWX geEjtSbzp12IcaTv0GaToqrn3qNJ.ME.NoiuXNebUaoMh0nlbH5YmlLj3hljHDkookKjPW2VbpDK R89p43uDmiHkOTVrkedfPbUGqGeiz81EItRDRHng85C2t4JJeRdc3nLIXhYbvwOUfsnleFmwaKMz rltEJz2YPihgughBC65uNPyycriyQvhbJD03fsdfJzGRPnx2zRIwvsvpfmdm0mKXJiQ.Aqzme7gq 9tyNcqEAimdu_ceFGZ555izatr4c0uO_wgkypQyTWZsJC8DTPQvKTq6oOjjaR.X50zS5usI4spp5 eNAkNNgq3Kem8x5ixTXeF48n_5pFBPgamHD3ZV5PiCXkLOQvkf3Q5KEwUftLr99wrwSerA7LCb_0 GAp2zScBzKGhIHkqKfNVdCR6fjkFeNVofvYWLjVrKCNTU7tt8VUq1ORK6KB3HkjNHTtc2wnFp3cc d1leWsvdH6T1.ha6RcwxvI8IKsCYMRkhlPcV.7M6b4BrBBR4IhfDIMW.32tUNM9Q._BC_b3zaZ5Q ex..Pc1xYiS9h0zfVJnn0pZtiyM291Fhfdnce3Lda6r0jvFXiRBDQERVTfPOG4gqLXqGjfBYSQyY aRV4uE_SA8dBWQKHc2d8U6pw.DP5rqG5iUkOcM8qe0vAie9a9sGMxvZopQ4cl43iS91A8WDG4maH h0iKvQ9WYfR0mNQHN3GmfL6QwdD6OpLSyccissH4.be.pw9EkD99AfeaTeEzxIopJ7tssvDskSEC g2TYcD8VeJY205y3msbTI26eQ2LNiW20saXuWgxdfVKPf1.1THqO.QhrbFd3FEd9D6gsxBzCRMpI dsViqc6M5121QoVrHew1Yh9kfZcwhBdrQovkZ3cRtz4GpvWuZSMm07NS1Xzawp9favirQ.vjNORf FQbUYMFdyOVueqqRac7zDCB3.4kCM_eWKE3REZENS15NVtDBRyw5dUnev_49JrPDvO_ntvVFpUhW C0aair0OLokPW71Ue5l2cldw93pjxtCB.JRpTemPaQhQvT5ML9X9LSUtBlRdMs246RJLHYxxf1AH Cgg-- X-Sonic-MF: X-Sonic-ID: 4c7d09dc-6a88-4746-b093-562acf119da2 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Mon, 4 Sep 2023 02:54:23 +0000 Received: by hermes--production-bf1-865889d799-scr2n (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 49b8180b7d8a89330959be58be1b41d6; Mon, 04 Sep 2023 02:54:18 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: An attempted test of main's "git: 2ad756a6bbb3" "merge openzfs/zfs@95f71c019" that did not go as planned Message-Id: Date: Sun, 3 Sep 2023 19:54:05 -0700 To: dev-commits-src-main@freebsd.org, Current FreeBSD X-Mailer: Apple Mail (2.3731.700.6) References: X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.147:from]; BLOCKLISTDE_FAIL(0.00)[98.137.66.147:server fail]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RfCsF2xYwz4Yv7 ThreadRipper 1950X (32 hardware threads) doing bulk -J128 with USE_TMPFS=3Dno , no ALLOW_MAKE_JOBS , no ALLOW_MAKE_JOBS_PACKAGES , USB3 NVMe SSD storage/ZFS-boot-media, debug system build in use : [00:03:44] Building 34214 packages using up to 128 builders [00:03:44] Hit CTRL+t at any time to see build progress and stats [00:03:44] [01] [00:00:00] Builder starting [00:04:37] [01] [00:00:53] Builder started [00:04:37] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.6 [00:05:53] [01] [00:01:16] Finished ports-mgmt/pkg | pkg-1.20.6: Success [00:06:15] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 [00:06:15] [02] [00:00:00] Builder starting . . . [00:06:18] [128] [00:00:00] Builder starting [00:07:42] [01] [00:01:27] Finished print/indexinfo | indexinfo-0.3.1: = Success [00:07:45] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22_1 [00:18:45] [01] [00:11:00] Finished devel/gettext-runtime | = gettext-runtime-0.22_1: Success [00:19:06] [01] [00:00:00] Building devel/gmake | gmake-4.3_2 [00:24:13] [01] [00:05:07] Finished devel/gmake | gmake-4.3_2: Success [00:24:39] [01] [00:00:00] Building devel/libtextstyle | = libtextstyle-0.22 [00:31:08] [125] [00:24:50] Builder started [00:31:08] [125] [00:00:00] Building print/t1utils | t1utils-1.32 [00:31:15] [33] [00:25:00] Builder started [00:31:15] [81] [00:24:59] Builder started [00:31:15] [33] [00:00:00] Building databases/xapian-core | = xapian-core-1.4.23,1 [00:31:15] [13] [00:25:00] Builder started [00:31:15] [81] [00:00:00] Building devel/bmake | bmake-20230723 [00:31:15] [13] [00:00:00] Building devel/evdev-proto | evdev-proto-5.8 [00:31:16] [41] [00:25:00] Builder started [00:31:16] [41] [00:00:00] Building devel/pcre | pcre-8.45_3 . . . (Looks like lang/go120 ignores the lack of ALLOW_MAKE_JOBS . There may be others that still have signficant parallel activity.) [main-amd64-bulk_a-default] [2023-09-03_13h48m45s] [parallel_build:] = Queued: 34588 Built: 727 Failed: 1 Skipped: 40 Ignored: 335 = Fetched: 0 Tobuild: 33485 Time: 01:36:51 (So about 1 hr after the last "Builder starting" it had built 727.) The vast majority of the time: lots of cpdup's with tx->tx showing most of the time for STATE but showing having some CPU time. ^T commonly showed various Builders in starting PHASE for 3min..6min. Around 66% mean Idle time (guess from watching top). After ^C "gstat -spod" reports it is almost always writing 2200 to 2500 writes per second or so for *hours* (still going on). ztop reports 1500 to 3200 d/s or so almost always for Dataset zamd64/poudriere/data/.m instead (also still going on). Normally no other Dataset is shown. With all the disk I/O activity, this is definitely "live" in some sense. But I've no clue if it is just repeating itself over and over vs. if it making some sort of progress. For reference for the ^C and after: ^C[01:39:00] [20] [00:00:03] Building sysutils/linux-c7-dosfstools | = linux-c7-dosfstools-3.0.20 [01:39:00] [93] [00:07:12] Finished science/dimod | dimod-0.12.11: = Success [01:39:00] Error: Signal SIGINT caught, cleaning up and exiting [01:39:02] [63] [00:06:34] Finished archivers/unarj | unarj-2.65_2: = Success [01:39:03] [128] [00:07:47] Finished sysutils/shuf | shuf-3.0: Success [01:39:04] [113] [00:07:06] Finished devel/bsddialog | bsddialog-0.4.1: = Success [main-amd64-bulk_a-default] [2023-09-03_13h48m45s] [sigint:] Queued: = 34588 Built: 752 Failed: 1 Skipped: 40 Ignored: 335 Fetched: = 0 Tobuild: 33460 Time: 01:38:56 [01:39:06] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-09-03_1= 3h48m45s [01:39:14] [12] [00:09:07] Finished archivers/rzip | rzip-2.1_1: Success [01:39:14] Cleaning up exit: cannot open ./var/run/49_nohang.pid: No such file or directory exit: cannot open ./var/run/87_nohang.pid: No such file or directory After that ^t produced the likes of: load: 6.39 cmd: sh 4849 [tx->tx_quiesce_done_cv] 10047.33r 0.51u = 121.32s 1% 13004k #0 0xffffffff80b6f103 at mi_switch+0x173 #1 0xffffffff80bc0f24 at sleepq_switch+0x104 #2 0xffffffff80aec4c5 at _cv_wait+0x165 #3 0xffffffff82aba365 at txg_wait_open+0xf5 #4 0xffffffff82a11b81 at dmu_free_long_range+0x151 #5 0xffffffff829a87d2 at zfs_rmnode+0x72 #6 0xffffffff829b658d at zfs_freebsd_reclaim+0x3d #7 0xffffffff8113a495 at VOP_RECLAIM_APV+0x35 #8 0xffffffff80c5a7d9 at vgonel+0x3a9 #9 0xffffffff80c5af7f at vrecycle+0x3f #10 0xffffffff829b643e at zfs_freebsd_inactive+0x4e #11 0xffffffff80c598cf at vinactivef+0xbf #12 0xffffffff80c590da at vput_final+0x2aa #13 0xffffffff80c68886 at kern_funlinkat+0x2f6 #14 0xffffffff80c68588 at sys_unlink+0x28 #15 0xffffffff8106323f at amd64_syscall+0x14f #16 0xffffffff8103512b at fast_syscall_common+0xf8 The console/logs do report "witness exhausted": . . . Sep 3 13:41:08 amd64-ZFS login[1751]: ROOT LOGIN (root) ON ttyv0 Sep 3 13:51:35 amd64-ZFS kernel: witness_lock_list_get: witness = exhausted Sep 3 14:26:38 amd64-ZFS kernel: pid 27418 (conftest), jid 245, uid 0: = exited on signal 11 (core dumped) . . . So it did not take long for the "witness exhausted" to happen. # uname -apKU FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #74 = main-n265143-525bc87f54f2-dirty: Sun Sep 3 13:35:04 PDT 2023 = root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-dbg-clang/usr/main-src/amd64.amd= 64/sys/GENERIC-DBG amd64 amd64 1500000 1500000 Looks like I'll be forcing the machine to reboot or to power off. The media was deliberately set up for doing risky tests. It is not my normal environment. =3D=3D=3D Mark Millard marklmi at yahoo.com