From nobody Sun Apr 30 02:44:13 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q89fM46grz48c32 for ; Sun, 30 Apr 2023 02:44:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q89fL5B94z4FJ2 for ; Sun, 30 Apr 2023 02:44:26 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=s0flWfP4; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682822665; bh=Dg1zRP6K+BkPLSZIDNaWBC+aEEpxAZJKuTftAcqwykQ=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=s0flWfP4B9WE88d56JpOwM21gfvfnRN+Jnkve3V3JuQam+1cFDlHx1uUxTnbrhcm+9AJb3oVkzHN9aIO3cwYZlrx2qx6OM50MhEjCccLLGHuTPorsVB0tkzjyG9+nq+hoNp9bHrJBuK6RrByBdQwJNEtzmDUtYj3yPL8gtzeGWPqfNRxXeuGTM331OYbglut8Gc1ZRHbG+UBLyc9PeG3HSG4oAwpfBcrX/5DnqG7xTEs3B6J4hbbpQgS3hLB2qc9PyO9doeCwIyOU7afSypHMlPXuyAw6QcKy1Yj10yXrsOExL7wIyxGWoLOpQwg5Q42M9HgDUkMK8wSpy0hGwiVIQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682822665; bh=nd5Ii+dBXUB/ugS8bU8qUae5jVgyBEn1AL++8gTUnta=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=cWxeM5J23bZN7CSYnlcGkq4wIFvPdRlFeABqJvyC8NWp5iYjYfoQAZHmYNbvUWjeEJ0IObn2Ig7fRPcpx+n6DfyujnQPI7PH9rGC1SkWaXfNScXR9c3xP2Od7XkFMt4eso6CS1bPih8N6+wFqNyUcscPWxnfcPhabA/C9daX4VviY2cAhZYcRog8q4W/XZ4uUCKNUwLvyoQF4D41opO4ab6nOaP0v6iz5j1CTs+OnZz1z2mRAneQGFM1btvs8hDhDVQFZBUXSHrPIaCIzcLrKoYYMlkq1mG268a4lOkl5zFvH7SsHaC+kGVupN7nKydnmMI3j3l1p++6hu8Dd0JoFA== X-YMail-OSG: WVuzG08VM1md.qqGM.tX2xSZ0snDT4G9Xsp6Lxrk_hed5Ud7wTtkNBm2wVIVmVx 0TZqzngtAmmloe1JSFVvZokTb53JBd7xCqJ539XlSF7ifXayryJvvC2IaC3T9apRiUEj15piK3U4 JhrbRFm_m0OBK8.Lg1wNTpycC_IYbbTNRC4X8KspLhmGKp4.hXvCTyrH02cOftrrH67VqeNfXo2V Xwi.m8uSQntmVepI3O857EroCqXXSeMtsQhDxzK7koa8lfICYYtJUpjSNY2F8sKJIph624RQGKv3 nQPM5bCJqn8dWQfWNtF40QX9CsuwGe5XELhGEfhGwfTraGz.LM.NsaDNd5CQ0l4UI1HFIfwIzhou vuKt78CgvkAnlYMYcg2n08Zu7orj.0b_6iaED1vANu97Nn6m7fkf6rSI9WpfA3ktRbQ_7oneEBeA N912iAmMANH7n2DYRHWCfaefqUPynrTCjiLTJTdXszz327oLiOJ1AckYzav1daAhidljjWJi18Xa FIp7gIjb.261mcFgEy7lpPimtlf1C8rmrFwjeFul_SWsV4Ds5dS7dcRHt6EykJ7KvgNhtf0qQIyE nLh6Vw2Xf782KX8_kcTgw5I35lXQECqIKDcLAFQohEZszHu0nH3FxxORhsF06UOyaZDxwx_Vu3tc 6iDgnDw.SLHih7sxoarYyywQfPaLto_TwHmzGSmtSFyruSzIFj4zRNDwcLGkatM2cwG_k.QU0gEQ U3KtrqGSaV6kC4tHgKnq67sKQkykuDqtgYo07x9mEWS6ul.6ExuR8GGh4yesM0JZseEOlPUi2idU df0K9Ynca9geYTu2woPXPox21nE2q.mjfT3FUpcQLcAFF5szJo0BmnBgZRYR5dFOfhWJHqQr_LNc _KO_eS7.tpShej0bDpScbaqYDqPITdgc3fQJBb9VcAZYP.UrtW36a8ug7F0OFcrbDggiABL1HUPN kGvzgy3A0N_0EJJI14HW04ro0yZEcUqnkK.KH.GyueKfQkmjaL.fcBEgN6Vva_dl0tMCzAXKG7UP xPMue.B.QFjSOUFqnruQmcNLcbFw14Z23yWlORXraXPyxPlpUd8ofxLn1QK7UwF5ZaP.Ul9xq21u hEtxNrUWqv8r32nYqEMP_d1GIgxVoPXp1g7z97bzOZrB9eG0q0xUz_prkwRkBueG22unZJMhmXJ4 qfjsivJlERgJ2xNg8i4MfEhMWLU4RsHnS2jiNafFnkmyizL7oSj2PaP6nhB6ES1NPwKJPWsnx897 GIvaVF4OPJkM.4qgCx6TbM97QAXMVeyTIp.hK7e3XcMIqghp8XT0TWEJ_9ykC8XFjdK8nEvSkhk4 Qzcb0lgCcVNUBBtqMCv2sdSFWi5I96WXh7PrGOgUrh8Adbi7zxN7GxxB_R_gGq2oWSkEMbeMRt.z McN.U6Zn80cn3su0ijyXJnGbFKr47Kt5hwI1JCc7udBCNV9HIiQMhqMyQA.S4a9O995dhrlLVknz oMM7J9nCHGbUJ.LGEc9DKpcsf8SUxQSG_KOC2bhOqFhje8idH.NXoGyiNpbLegOtp5ozoAi59uWf Kz1tzjFEnmkqcBdOxQ6.laaYolcQJM5MSJA4O5MPCuXk8nrMRTU_XXEeZYpfHFiYrTHkRq4i_vAQ _HPgJ5V8ze6p82PRMnXD5V2HvE7_rfqazu3Ry813gXcLRVCQmoMNbpVU68.6S.XzqpZDqyUOI6Wa YhQ4IJsKG.5hmOxgEpxKJ2hbXzOnVD2EQ7jG2nz1.h5U_sauRKndCaUvb7WDkeXOqkORZmsTpVec 1eU1R1ri75g1tsUvgSVd0gYTSFGW2YkHwRnQEKjqDJK5Ka_.jwfOgkGT9g9qR7hLHQvx63LkUeqC kFmLJUs7I8a1KE9HeFM90gVQVdHkrVQc9tZh3hibEkXjVZAyAdikaHcj5siC8.coSnH.dSaQtbtk ik9KpcaQ.apO1x_huFuZ176RJqqCS78ZTvXHGglsl2uSQElnbAuQTrO72KUgkUfICGID6lEEiG9B vW1RVp9H2fdCv2uYcKme46ryZzuFtQcVCkyT3jRXKy97ZnOJG.7h5RwelCr8G50TPjezC5Jmkvav _Q_AUghz8r_BH5qLbkF_t6tY26EUnFg77ec.JeqY572lbziWPghnRfs9dIkdCM5.P0RU1I4h3PHK wEtnGwiCIfqXfhZH7XcBVnQZ5wK_uUCi_ksvVL43vhUstIu_irZfQNRoz7bRibP4Q9wVgeS9rTFT oYI7xdcCvHP2XJSC2p94IDFfuCMzefmT5HS9H7yHPeASCTKIjo6QgKRgXOdwrVvWuy7m6oJCqzLF m X-Sonic-MF: X-Sonic-ID: 600319dc-fa0d-4f7c-9d89-4504435516d5 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Sun, 30 Apr 2023 02:44:25 +0000 Received: by hermes--production-gq1-546798879c-g88f5 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1023595f13d6ecaf5c33461fb2a882a1; Sun, 30 Apr 2023 02:44:24 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? Message-Id: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> Date: Sat, 29 Apr 2023 19:44:13 -0700 To: FreeBSD Hackers , freebsd-arm X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D.ref@yahoo.com> X-Spamd-Result: default: False [-2.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.996]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.147:from] X-Rspamd-Queue-Id: 4Q89fL5B94z4FJ2 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N This is based on: main-n262658-b347c2284603-dirty, b347c2284603 being from late Apr 28, 2023 UTC. (The "-dirty" is from some historical patches that I use.) The build is a non-debug build (but with symbols not stripped). World or Kernel had been built using: -mcpu=3Dcortex-a78C+flagm+nofp16fml just for testing purposes. (Worked nicely for -j8 buildworld buildkernel testing for the 4 cortex-a78c's plus 4 cortex-x1c's present.) Monitoring poudriere bulk related activity via top and gstat -spod I see a lot of the odd result of one process doing something like: CPU4 4 1:39 99.12% /usr/local/sbin/pkg-static create -f tzst -r = /wrkdirs/usr/ports/devel/cmake-core/work/stage while other processes sit in the likes of: tx->tx zcq->z zcw->z zilog- select wait But sometimes there is no CPU bound process and the top CPU process is the likes of: 1.24% [usb{usbus0}] "gstat -spod" basically shows da0 dedicated to write activity most of the time. After: sysctl kern.tty_info_kstacks=3D1 Then using ^T at various times, I see a lot of: load: 0.48 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7534.91r 11.06u = 22.66s 0% 3800k #0 0xffff0000004fd564 at mi_switch+0x104 #1 0xffff000000463f40 at _cv_wait+0x120 #2 0xffff00000153fa34 at txg_wait_open+0xf4 #3 0xffff0000014a40bc at dmu_free_long_range+0x17c #4 0xffff000001448254 at zfs_rmnode+0x64 #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 #10 0xffff0000005fc430 at vinactivef+0x180 #11 0xffff0000005fba50 at vput_final+0x200 #12 0xffff00000060c4d0 at kern_funlinkat+0x320 #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 #14 0xffff0000008f8514 at do_el0_sync+0x594 #15 0xffff0000008d4904 at handle_el0_sync+0x40 load: 0.34 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7566.69r 11.06u = 22.66s 0% 3800k #0 0xffff0000004fd564 at mi_switch+0x104 #1 0xffff000000463f40 at _cv_wait+0x120 #2 0xffff00000153fa34 at txg_wait_open+0xf4 #3 0xffff0000014a40bc at dmu_free_long_range+0x17c #4 0xffff000001448254 at zfs_rmnode+0x64 #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c #7 0xffff0000005fd6c0 at vgonel+0x450 #8 0xffff0000005fde7c at vrecycle+0x9c #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 #10 0xffff0000005fc430 at vinactivef+0x180 #11 0xffff0000005fba50 at vput_final+0x200 #12 0xffff00000060c4d0 at kern_funlinkat+0x320 #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 #14 0xffff0000008f8514 at do_el0_sync+0x594 #15 0xffff0000008d4904 at handle_el0_sync+0x40 load: 0.44 cmd: sh 93914 [tx->tx_quiesce_done_cv] 7693.52r 11.24u = 23.08s 0% 3800k #0 0xffff0000004fd564 at mi_switch+0x104 #1 0xffff000000463f40 at _cv_wait+0x120 #2 0xffff00000153fa34 at txg_wait_open+0xf4 #3 0xffff0000014a40bc at dmu_free_long_range+0x17c #4 0xffff000001448254 at zfs_rmnode+0x64 #5 0xffff0000014557c4 at zfs_freebsd_reclaim+0x34 #6 0xffff000000a1340c at VOP_RECLAIM_APV+0x2c #7 0xffff0000005fd6c0 at vgonel+0x450 #8 0xffff0000005fde7c at vrecycle+0x9c #9 0xffff000001455678 at zfs_freebsd_inactive+0x48 #10 0xffff0000005fc430 at vinactivef+0x180 #11 0xffff0000005fba50 at vput_final+0x200 #12 0xffff00000060c4d0 at kern_funlinkat+0x320 #13 0xffff00015d6cbbf4 at filemon_wrapper_unlink+0x14 #14 0xffff0000008f8514 at do_el0_sync+0x594 #15 0xffff0000008d4904 at handle_el0_sync+0x40 The system (Windows Dev Kit 2023) has 32 GiBytes of RAM. Example output from a top modified to show some "Max[imum]Obs[erved]" information: last pid: 17198; load averages: 0.33, 0.58, 1.06 MaxObs: 15.49, = 8.73, 5.75 = up 0+20:48:10 19:14:49 426 threads: 9 running, 394 sleeping, 1 stopped, 22 waiting, 50 = MaxObsRunning CPU: 0.0% user, 0.0% nice, 0.2% system, 0.1% interrupt, 99.7% idle Mem: 282760Ki Active, 7716Mi Inact, 23192Ki Laundry, 22444Mi Wired, = 2780Ki Buf, 848840Ki Free, 2278Mi MaxObsActive, 22444Mi MaxObsWired, = 22752Mi MaxObs(Act+Wir+Lndry) ARC: 11359Mi Total, 3375Mi MFU, 5900Mi MRU, 993Mi Anon, 93076Ki Header, = 992Mi Other 8276Mi Compressed, 19727Mi Uncompressed, 2.38:1 Ratio Swap: 120832Mi Total, 120832Mi Free, 2301Mi MaxObs(Act+Lndry+SwapUsed), = 22752Mi MaxObs(Act+Wir+Lndry+SwapUsed) The poudriere bulk has 8 builders but has ALLOW_MAKE_JOBS=3Dyes without any explicit settings for the likes of MAKE_JOBS_NUMBER . So it is a configuration that allows a high load average compared to the number of hardware threads (here: cores) in the system. I've rebooted to do a test with filemon not loaded at the time (here it was loaded from prior buildworld buildkernel activity). We will see if it still ends up with such problems. =3D=3D=3D Mark Millard marklmi at yahoo.com