From nobody Sat Mar 25 22:35:36 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkYnj6C2Kz41pj9 for ; Sat, 25 Mar 2023 22:35:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkYnj1FYkz48sS for ; Sat, 25 Mar 2023 22:35:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679783751; bh=Yol58pV/rLr38QiELsLObJTdfgFajduQyOavu7zNkQc=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Rjzm2WWqTzHhOTt1vX0FSnipOsUoQNkNNSjVnT1m8zqH/+jo8bvCnlSw3x8gTOpSIY8bQNOt/zV8NQ4cmY6GiUeAA0nbq2HgikL9hFrhfkjHlSFDKaVNVFsMSg++SmVKxb4GFigi1GAVAbhaU0kk283aQNhPbqZx0YEEa05yTNh2ed4j1u3SBkil4263nM5wNBTiLZAd6zmdnMhigIQIhz9we2SH7/HmMe3fulm4ChM264gcWBaonqo+njFjHWWTGyELmBAkeYTbxveZh0eKKM32EkHAwXl4yTI1bkc8ixJ+TAckSDQzOQlnGMeRBUVgLMBlWscYD/VsBy9iC/7/jg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679783751; bh=suhqWNlWwL+8P8iU2V9Ny0PD7NTSxrB3uaflPreqD3b=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=H52vfciWKnC0VEhaK0vuSRtcd1haXKIcECnSsYUHU9QGqJRS1aijEwVfelKP2lJUQ32Yaa/ERh1B8acZOCMtSLXLq/p4lx39EXcpEJ18QAJPk/kkkbFsT7uowaboyt6Nv/3+y82Ng0Z35/vJBnzOdf70JZ2DXAY0qotVNZFc9kM7D239mFcA69tCrR2FadNS/99nUII3PzRqGMqYVIzwmg+9sBBWPT9Tf4G12qTgmKl0UkA7ngvySxNZ8mli426TGX+1FZkpku/uZrovHp96eKzd1UAogiqVZBmtkpVefbifhJ0Zrm4h5pIKsAIv5xYPbMz/B07Rh1neFhl7qDT9+g== X-YMail-OSG: ZVDXDiMVM1nWjjAnJ.hytJcoAHL_BKqHcgaZuZrYEStf5Z1ubVMsgNVD95Af9G0 SdpfClazpFldzePAW5o0uDM3FA7ccpbY3Fmxi0dh_JUiNMzGZPG4VdnsxvlnsoQhueY6HymcnUs3 dpc7pY1yEVbr1TCACvUTFg9BV9P1Ez_wThse96G7FYgBsRgUxdGuxe4WNGZthVhR4lE.fHkpL1kw IuOH9Lfe48mvZwOoUiwPNajxSFFfLaL_4kjdI0xKR.9qHVFBSEqjjOPhYJCZOCNczpU_X.5hrbqx 7SW.oH9v6bgYh0zZcniTUB.8xn.5OPb_ccepGLdeAHaDgvN4t4XSeYK..t69g0JWVTWAT5uFNCLO ZjHOz35AsLvXNlD4LswTglV3GLuBoFjI5o2XR_bXK6Ic26sG.m9_t065tXoUcX3tAjw_OQp6cm_D v7.KyrNcvIS_PslrV7a0kZIfXQaBurwbratiCNcYbbMU5J4MEXpCIkZE.CJKCU4AYiIwPDsuO6Cx vG0AFQUfVsx01cOLJqzu3wcQM4G0PSQTLrcWv7KZptHSONsXw_IqBgN5R.qlYRHzkiTl0rD9sR4M rTj40882FQfUhBEmUoakaatCj8e0vxJ7smVTkWCF4lca7kZV.Hwpm8hNp3iXl9ChIABuI9fOtZ83 RhmVV7LTBiENFkelAyJyCMqg4ofwZsg6mfTKwjUzmGaXN92lm5OY_IznclHM.ri4mYZDDmLIwxX0 yTg9r9ZKcTrpPY7rhsHlB4W0.jbrN.CbmmgWB4f3qzwAaWlp9fu_oM8EEj1R7L42skmtdswM7iJq iWlng2mX7DUc_mskSTnrfbh5RCiLIW5hhyQ0iuoKZS07kULCyUbiFsYJErCeyViMpZ5K.ANhhxa_ RIOdinu3C6LEWALAREC.N0L3A_YiH9w1aKj8X5V4TLt2hpJqZ02w9EaST6W87I_yAIIlri5ge2Si EYGzhLTAx2i8QFDOtkTSt91WNcem72Bwdp3JxykhLzyD5KWWYTR5LSvm7LHiDswqc6cRbN0Upad6 DaHtYUyXxhOYK3XbXi03DuCfKbsKiSNyPJiM_liKGtNR_2i7TOavTixcKTxGNQhYvXErIBitIe8h lXKBagOwrkzE.Cm5ipDCPtjeu.NmEumNDLCRPLlE2ZrQve.f1QN7rCh_1JV_mh4fVRs6AdztlD72 L9ZHGJxiJaSKc4JrP528eMC3OBkBCuzsr3M5y8S2GotkXJO3o.JTedzEDXDFZFw7OxECI0GtAtcY 98jougNa7nOrIjYG9RQ1hiMdQSImokYLaeZXwkcjhTTqw9YJeu4JrDvaGFaU8wiGzQ6X5rvfvy4U iJPANe1d8bXsOOkAx1DL6P83LKsU7BzbXu0IRxtDHu26Py7vaGmyyFqGQDWC0r2CD_QQdNzqx8Oe 47HSmADAjPJl8mMwDEeS2oDA5DZpV5wJlsH3aUm2rZ0.wKpb_AYY.TQWPgfGryjufxBMPj8mFnE5 P2czRD2kakODylTJRxf9G99BGMgUmr_GkZZA.csT7X7d12pmRy.N8dUaKLBTN3edM7EH7nLV4I_M KDogzAzyyi85xT.xwEdxy9mWnqZmfy.PWAMySBkYoUNjPnKaZ4IE_CtPKEuUrPSR5wc4L_x0DC1x UgAdRSpR4Zi9kjw7.lNCaR2I9Hz0MljchQ0BMyUEWJYFEK3AEn_PnP3vgZiHt0E7qHvLrFRADLJo ngOMH3BrCMnDvR7L1E85XkGGrNQIA_Grm5EQ8VeMQ8fCqucEBWtqrKF5fWCGai5uInOTesHkcjAv reCWxxJ1vVfdlPwLh8ydol01X3LZ.m4TzuJiBi8FMtqYatc_94x4fX07jI6SnZ9Ac0HhVjV3jRKQ gUd8mLlUC_sAwIqf6vnjHnuURh5VDG29IZhkOJGkrPf9KPrd2zqQYUhvxnzhJUG.IWDavC2IvmUU q7dRQ.MgS.W2XziYmBZC_Ldkgx1fs6JjX7JA9C1GHYUgjwntt2aQrjtvru7SAp_DYfFSCWPwXvsZ fYORPvvlG1ULjcHA080DD8Cj_gPn7quvMFqnWHkf5v.to9.iRAuOjkcz4aVDNjIb7rHmFtShhHVJ ixq5DINQOQwDMdcncOzwmn4b5k3ZT4PJrFyPlOmwQvtHoL6l39SGL95C_ctT2RxnQiSMhKgPFyaV 8ijiiXzgkQsASVq9ixDgPgRrxE8G80YRbxyXRjA7nY58OwT7P9ieYgcwiH36d.ZYNI.gnup6CwsI YcwuJKl_MrQT_1VjtjKj0ZVoABjiVhSmwwplA8xSD7dfrNavfvDFDo0aCtc.D1fjCNVVoyLnalc_ VSI_pTlWIkuM- X-Sonic-MF: X-Sonic-ID: 92a80676-41ad-4662-942c-061be1f69067 Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Sat, 25 Mar 2023 22:35:51 +0000 Received: by hermes--production-gq1-6cf7749bc8-g5z7v (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID e30ae660f78010d228b43b0325d0c153; Sat, 25 Mar 2023 22:35:46 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Sat, 25 Mar 2023 15:35:36 -0700 Cc: Mateusz Guzik Content-Transfer-Encoding: quoted-printable Message-Id: <21FEBFC6-7A43-43C6-A888-EC9962D738B8@yahoo.com> References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> To: Peter , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4PkYnj1FYkz48sS X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On Mar 25, 2023, at 14:51, Peter wrote: >=20 > On Sat, Mar 25, 2023 at 01:41:16PM -0700, Mark Millard wrote: > ! On Mar 25, 2023, at 11:58, Peter = wrote: >=20 > ! > !=20 > ! > ! At which point I get the likes of: > ! > !=20 > ! > ! 17129 root 1 68 0 14192Ki 3628Ki RUN 13 = 0:20 3.95% gzip -9 > ! > ! 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 = 0:00 0.27% tar cvf - / (bsdtar) > ! > ! 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 = 8:05 95.93% sh -c while true; do :; done > ! > !=20 > ! > ! up front. > ! >=20 > ! > Ah. So? To me this doesn't look good. If both jobs are runnable, = they > ! > should each get ~50%. > ! >=20 > ! > ! For reference, I also see the likes of the following from > ! > ! "gstat -spod" (it is a root on ZFS context with PCIe Optane = media): > ! >=20 > ! > So we might assume that indeed both jobs are runable, and the only > ! > significant difference is that one does system calls while the = other > ! > doesn't. > ! >=20 > ! > The point of this all is: identify the malfunction with the most > ! > simple usecase. (And for me here is a malfunction.) > ! > And then, obviousely, fix it. > !=20 > ! I tried the following that still involves pipe-io but avoids > ! file system I/O (so: simplifying even more): > !=20 > ! cat /dev/random | cpuset -l 13 gzip -9 >/dev/null 2>&1 > !=20 > ! mixed with: > !=20 > ! cpuset -l 13 sh -c "while true; do :; done" & > !=20 > ! So far what I've observed is just the likes of: > !=20 > ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 2:03 = 53.15% sh -c while true; do :; done > ! 17735 root 1 111 0 14192Ki 3676Ki CPU13 13 2:20 = 46.84% gzip -9 > ! 17734 root 1 23 0 12704Ki 2364Ki pipewr 24 0:14 = 4.81% cat /dev/random > !=20 > ! Simplifying this much seems to get a different result. >=20 > Okay, then you have simplified too much and the malfunction is not > visible anymore. >=20 > ! Pipe I/O of itself does not appear to lead to the > ! behavior you are worried about. >=20 > How many bytes does /dev/random deliver in a single read() ? >=20 > ! Trying cat /dev/zero instead ends up similar: > !=20 > ! 17778 root 1 111 0 14192Ki 3672Ki CPU13 13 0:20 = 51.11% gzip -9 > ! 17777 root 1 24 0 12704Ki 2364Ki pipewr 30 0:02 = 5.77% cat /dev/zero > ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 6:36 = 48.89% sh -c while true; do :; done > !=20 > ! It seems that, compared to using tar and a file system, there > ! is some significant difference in context that leads to the > ! behavioral difference. It would probably be of interest to know > ! what the distinction(s) are in order to have a clue how to > ! interpret the results. >=20 > I can tell you: > With tar, tar can likely not output data from more than one input > file in a single output write(). So, when reading big files, we > get probably 16k or more per system call over the pipe. But if the > files are significantly smaller than that (e.g. in /usr/include), > then we get gzip doing more system calls per time unit. And that > makes a difference, because a system call goes into the scheduler > and reschedules the thread. >=20 > This 95% vs. 5% imbalance is the actual problem that has to be > addressed, because this is not suitable for me, I cannot wait for my > tasks starving along at a tenth of the expected compute only because > some number crunching does also run on the core. >=20 > Now, reading from /dev/random cannot reproduce it. Reading from > tar can reproduce it under certain conditions - and that is all that > is needed. The suggestion that the size of the transfers into the first pipe matters is back up by experiments with the likes of: dd if=3D/dev/zero bs=3D128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D132 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D133 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D192 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D1k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D4k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/zero bs=3D16k | cpuset -l 13 gzip -9 >/dev/null 2>&1 & (just examples) as what is paired up with: cpuset -l 13 sh -c "while true; do :; done" & Such avoids the uncontrolled variability in use of tar against a file system. But an interesting comparison/contrast results from, for example: dd if=3D/dev/zero bs=3D128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & vs. dd if=3D/dev/random bs=3D128 | cpuset -l 13 gzip -9 >/dev/null 2>&1 & as what is paired with the: cpuset -l 13 sh -c "while true; do :; done" = & At least in my context, the /dev/zero one ends up with: 18251 root 1 68 0 14192Ki 3676Ki RUN 13 0:02 = 1.07% gzip -9 18250 root 1 20 0 12820Ki 2484Ki pipewr 29 0:02 = 1.00% dd if=3D/dev/zero bs=3D128 18177 root 1 135 0 13364Ki 3048Ki CPU13 13 14:47 = 98.93% sh -c while true; do :; done but the /dev/random one ends up with: 18253 root 1 108 0 14192Ki 3676Ki CPU13 13 0:09 = 50.74% gzip -9 18252 root 1 36 0 12820Ki 2488Ki pipewr 30 0:03 = 16.96% dd if=3D/dev/random bs=3D128 18177 root 1 115 0 13364Ki 3048Ki RUN 13 15:45 = 49.26% sh -c while true; do :; done It appears that the CPU time (or more) for the dd feeding the first pipe matters for the overall result, not just the bs=3D value used. =3D=3D=3D Mark Millard marklmi at yahoo.com