From nobody Thu Mar 23 02:44:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhqS5754wz40PqX for ; Thu, 23 Mar 2023 02:44:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhqS45qQqz4Jp5 for ; Thu, 23 Mar 2023 02:44:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="Crknf/8b"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679539474; bh=JbEpmCgl8zYC032jZdGYKfLeTuYbtTuRj8c1NK61dvY=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Crknf/8bZsU04m3OcpoPapvFFYqgJsXlxY0JZvLydWR1mEe07WgMZIOVzKgTkUx0FPGaVK0AE2vrR+/XN5MMBSU6iVkbh0jxJSVcLdOhucCprLRqHciBdRhvFNy1SOvGKIS210N0HYYq7sxPbU+n3Pc9mCcZAVyW3HB5hlLszqfh7sejWRVthViAKowpL3UtFtDs2DRXd5CPBBk6YYs4AGC4LRxKddHCqdVqSwJ8g3Xlax2dbOGMG4x2qDAFf2ssV6E65TgzfGFaGuUPhYMyKeN6PyWS53rKb4PJHciKvSgnVLOkZwxtu076MvwaqmVh5g7QBBrqBmrkIj/T4wtWVQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679539474; bh=AhrrrEB9nzXzBTarAJAvNBl1q19KBdarSa0U1WvkCYs=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=uNU6ErwfNXJ4/0oeEBHgi+IPgwMPNjI3hiSP7qh/3Hy4El6Yhdn5uuLlMU2zXjf7OHagDFhJDrvJe1lrLoOts9H3okWxqbGdr6dww73Areh7ZzXluEcpmR115QMcx4tiy5OyTBvhvgoSYH7CqUYdAe+uEHYVmTENgfznE/lOx3vaKCy9wy2lpXZOjsVz44g+FmSTWsgxMZOSyj4bm8+g1E3e+3LIzTMG0gSpe6qBY16gE+onP8aIiZzqTH6CJwMpSwS0NaCVkWgOIIELd1tbdd7/Hlb8LV1U4AmEoX4KrH95W7b6VKdcDKQ44wkh0yr5Yig5goGtYucPFeznB6ibJw== X-YMail-OSG: sDEO51YVM1mA0PLPVyiGpj2Hn6Usod3C_6srxem57mIWMhxzX93t3.Uoyxg8O3P E6qBGLl8Tfj9jc8v0nezJYggt4Wqhiy7LIVIXIgmhDOf4JpZt2wbqphp9ckxm1rVxByUThmiYDS4 sezmmCxVh5d3ZIPKYtrDDCPxHl88KH5vKaaurP8pkw73HhBKNLM7D5n0aFwbpLYB0x9JtURF0LuH QfWDAKQR4Z3mvmzRdRn61jRbALAOouaa7FBgpEA9JShiD6GCUFdeimvBr_cV0Dy2.jbXUy_MMfJH C0z6xR55kCwWgLfNRIlOOh_9E_EJbWdqaO7Lx_E.bEy41GX9YO3SG8sB2o3y3t4D1GOR_wA2ENTR ev0Pk7TtZTRVR5LIItnw0_4QZWq4V5r7AzCbvcC_Wd1qdzTqIfwq7FbFJk2iml0oAmF9Td940rVp 5ylgzZV7w2UutmW_nlW1KgxU_M5IZ.lYj3q6YxlLCAIE78o5hZiwsu6bCM3qr8ukxiGypnoXWVhm 7hP9KsuCjAVpU.WofcVL_FR7zsasWsCuNEYSWO_iEQqqEkNRQvG6VIt2UA3aII1jNQeXiYFtiUeq 6WcF5s43DphHIodgjiibCB3o1omZs9Zm9dbIGdftrNP6ie4GjNXaC17g.ud4T3QD_0xoHcHKfOx0 ylQUmwWb0KUvtKkgxx6WMMxCsBcCE106E85GcPDTode2TKEB9C5jGZ4xO2PcAYx.dDKzn08AHAFo xy5rRxZJ0zwIeSFwxTHky_RUGTsYlLsJwIg6kZR8f0nrPsXUasBvLAd_pTJRgko4FR5Ey6z5w5x9 8Qodc7yZVOnHdBJ2m4sG6eDD_skpfEgs6w252hoL6oc4xTjLoB8VILlIN69qeF1rumB.z7INA0XN .lLX3h1fEW.avuK_RlpKeg.SQXL9ebLpWVKiZ9sIqGtX6TWgMbbaTVqEHWn6.rIEL4bfImAY.mbK jXGIbQX9teSlc1mYS32UDxuwpUAkAKBvqMEsbcmNls35e25DesLwcPpIlRJS8gGVZXuZ_nhzP.cr vYtcyrSlVHd_.1QjUGFLvtO2IQqll.HFHdXfWHOcdqjV6d5GATC_jGVXKStpb_fLV8d70K0pxA3h HH.v9bxeIUwUdeMxApOEyi29_9rQ9BKh49txsE.pjWLoG9Z1kIW_QJO3Bs5juZZH_I.LJKGM3QOc DX60ixROY3IZel4pLQ0WkYlgU_8fIu34_8rpeSEi4kAwbhsm2FBmxlrvMjUQjRtqmxAM6gQmD9fe XtucUrsZE1oLYhsI52D6z9_7.FqTeCSmvfRlvV9p7wRQ4G3P8945bA9uZgJnZT2N0VMlcrQGVjWN NlpL3fXyCrmEaHGoezpoVYbvgiVfA0sYXj7Pr8uHCVQKRBqRdQ2ruK_zj8bBWiM29Bxb8N6a4p_k Gl8bZMarFizIQcuT8z.YMmbtbVCX3At49HsbkHKk0QfrQhby7PfJqpczGgUHj9pt_6U1Opq8RC6G 9MetLP5va2DTrAsE6KTIu9mhqX5CVWHIR6O946cTkt0qu05B.obAXgC6qnrSDNqq4mqaaHKCoIEh gDI9pjzCeL71M4yyVmnrT4x0DcsKdObsJMbMPoFUP6MzDkoDqqly8xMLXchUYo.Azfb0_K_dsvNU thJdomKZeZ7XvD6OfCSZXylqo9SuYkuJSq_DUhzS6bmEFBhqUg.jnl05EBikNxMXaBjwM92WLNey EzYFfb9EGSZMsED6jgEoh8b93jVuHhuzidvSzSimrlwUTgIFVMU0a9VrR9Gz_bQIKB.0y_ys3YpU vcizEP3Mu1TiQLcS0oZnYWFDXv0AG1ppOJJUXyzkgdNdV2OVDmKT6Ku4d3KzbBpH14igzRo11wVW 2Nsng2IXtmTPQS2lUyTFnSiWntrNm5AfmTPrKnicbe193VFQ5mm5YBKiKoWOFWMrVg2mvop3VdKS EyGTkWqSWN4qaVFoqvjOAj42rTujJUvggSisOe6CjzM1MCdGLnwkZTTavhxQTgAsWuyT.AtYIVfA rBh2ZihYlp5dQo8DVWNBqtix2ZGerb9Oz7jMYhxeDsPQCErgc70I8UOhivKH8xoUZkXu.te31G0y .mNHyKFtvi5G9T9aZERoAQw_eulel4QbuZXyilt3gFuOUJatDibgQDUaOeJ95AW.WYt9dMl3bTak lDAvgLDlgkSkJY2mLb1ohL21GMQjOg.MEG4Ry_15uEiSBaMdNxa51_UlahcNvkcYiWywQc04zX68 3PCRyYnOj1p7htoPCoIJbNBGYlT3shC8OO9K77cZiAaF071cu.TONkJUzLmj5vNb3PCH5Ikrj0WB vnLE- X-Sonic-MF: X-Sonic-ID: 03bf1ad8-89b0-47d3-8c48-66c78c7ce8a6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 02:44:34 +0000 Received: by hermes--production-ne1-759c9b8c64-fztnz (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 5662090b4cb4eb6cb040a3096caeea55; Thu, 23 Mar 2023 02:44:31 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: <0196A767-5DC2-4DA3-BA5E-62A358CBCF64@yahoo.com> Date: Wed, 22 Mar 2023 19:44:19 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> <0196A767-5DC2-4DA3-BA5E-62A358CBCF64@yahoo.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.148:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.148:from] X-Rspamd-Queue-Id: 4PhqS45qQqz4Jp5 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 22, 2023, at 18:08, Mark Millard wrote: > On Mar 22, 2023, at 18:03, Mark Millard wrote: >=20 >> On Mar 22, 2023, at 16:17, Mark Millard wrote: >>=20 >>> On Mar 22, 2023, at 15:39, Mark Millard wrote: >>>=20 >>>> On Mar 22, 2023, at 13:34, Mark Millard wrote: >>>>=20 >>>>> On Mar 22, 2023, at 12:40, George Mitchell = wrote: >>>>>=20 >>>>>> On 3/22/23 15:21, Mark Millard wrote: >>>>>>> George Mitchell wrote on >>>>>>> Date: Wed, 22 Mar 2023 17:36:39 UTC : >>>>>>> [...] >>>>>>>> Here are the very complicated instructions for reproducing the = problem: >>>>>>>> 1. Install and start misc/dnetc from ports. >>>>>>> Installing is likely easy, as likely would be building >>>>>>> with default options (if any). I know nothing about >>>>>>> starting misc/dnetc so that is research. (Possibly >>>>>>> trivial, although if it has alternatives to control >>>>>>> then I'd need to match that context too.) >>>>>>=20 >>>>>> service dnetc start >>>>>=20 >>>>> I built and installed misc/dnetc and got a binary >>>>> blob that clearly was not built in my environment: >>>>>=20 >>>>> # file /usr/local/distributed.net/dnetc >>>>> /usr/local/distributed.net/dnetc: ELF 64-bit LSB executable, = x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 10.1 = (1001515), FreeBSD-style, stripped >>>>>=20 >>>>> Way older FreeBSD vintage than the locally available toolchains >>>>> would normally build. Some might be cautious about such a thing. >>>>>=20 >>>>> The man page reported that: >>>>>=20 >>>>> QUOTE >>>>> If you have never run the client before, it will initiate the = menu-driven >>>>> configuration. Save and quit when done, the configuration file = will be >>>>> saved in the same directory as the client. Now, simply restart the >>>>> client. =46rom that point on it will use the saved configuration. >>>>> END QUOTE >>>>>=20 >>>>> I've not seen what the configuration asks about yet. >>>>=20 >>>> I went through the configuration, basically just looking >>>> at it, other than providing an E-mail address. Then . . . >>>>=20 >>>> $ sudo service dnetc start >>>> Password: >>>> Cannot 'start' dnetc. Set dnetc_enable to YES in /etc/rc.conf or = use 'onestart' instead of 'start'. >>>>=20 >>>> $ sudo service dnetc onestart >>>>=20 >>>> I just let it run without any extra competing activity, other >>>> than I had my patched version of top running. It records and >>>> reports various maximum-observed (MaxObs) figures, here >>>> the load averages being relevant. >>>>=20 >>>> Top showed that dnetc started 32 processes, one per hardware >>>> thread. Mostly I saw: 100% nice and 0% idle. >>>>=20 >>>> Letting it run and then looking at the load averages (and >>>> their matching MaxObs figures) after something like 60+ min >>>> (not carefully timed: was doing other things) showed: >>>>=20 >>>> load averages: 31.97, 31.88, 31.66 MaxObs: 32.12, 31.97, = 31.66 >>>>=20 >>>> (Note: The machine had been up for over 2.75 days before >>>> starting this and had not been building much of anything >>>> during that time.) >>>>=20 >>>> I've not yet experimented with having other, significant >>>> competing activity. >>>>=20 >>>>>>>> 2. Run "make buildworld". >>>>>>> So on the 32 hardware-thread (16 cores) amd64 machine that >>>>>>> I have access to, the test is to only have buildworld use >>>>>>> about one hardware thread, no matter what else is going on. >>>>>>> I never would have guessed that the steps would not involve >>>>>>> more like -j$(sysctl -n hw.ncpu) (so around -j32 in this >>>>>>> context). So it is good that you provided your note or >>>>>>> I'd not know if I'd done similarly or not when trying such. >>>>>>> [Note: -j1 and lack of -j are not strictly equivalent in >>>>>>> how make operates. As I remember, the distinction makes >>>>>>> a notable difference in the number of subprocesses created >>>>>>> directly by make (one per action "line" vs. one for the >>>>>>> whole block?). So even using -j1 might make a difference >>>>>>> vs. what you specified. I'd have to test to see.] >>>>>>=20 >>>>>> I am literally running "make buildworld" with no additional = options. >>>>>=20 >>>>> So required for repeating your results, but likely making >>>>> such results not be interesting relative to how I normally >>>>> deal with buildworld buildkernel and the likel, no matter >>>>> if there is other activity in an overlapping time frame or >>>>> not: my time preferences are too strong to wait for a single >>>>> hardware thread to do my normal builds, even with no >>>>> competing activity on the builder. >>>>>=20 >>>>>>>> Standard out conveniently reports how long it took (wall = clock). >>>>>>> But nothing in your instructions indicate about how >>>>>>> to get an idea much progress dnetc made during the >>>>>>> various tests? [...] >>>>>>=20 >>>>>> Honestly, I've never worried about this part. But dnetc logs its >>>>>> progress in /usr/local/distributed.net/dnetc.txt, though not in = terms >>>>>> that are easy to relate to real-world progress. Oddly, when I = run >>>>>> "make buildworld," I'm primarily interested in getting the world = built. >>>>>> Perhaps others feel differently. >>>>>=20 >>>>> Off topic for the specifics of the actual benchmark >>>>> that you run: >>>>>=20 >>>>> Then why not use of -jN ? In my context, any buildworld >>>>> using -j1 or no -j at all takes a huge amount of time >>>>> longer than letting it use all the hardware threads (or >>>>> so). (I've avoided having any I/O bound contexts for >>>>> such.) It does not take additional load on the system >>>>> for that to be true --including on the 4-core small arm >>>>> boards when I happen to buildworld on such (rare). >>>>>=20 >>>>>=20 >>>>>>> [...] >>>>>>> FYI: I've never built with and run the alternate >>>>>>> scheduler so if there is any appropriate background >>>>>>> for that that would not be obvious on finding basic >>>>>>> instructions, it would be appropriate to provide >>>>>>> such notes. >>>>>>> [...] >>>>>>=20 >>>>>> You have to build a new kernel, using a config file in which you = have >>>>>> replaced "options SCHED_ULE" with "options SCHED_4BSD". -- = George >>>>>=20 >>>>> Thanks for the notes. >>>>>=20 >>>>> I've not decided if I'll do anything with the binary >>>>> blob or not. >>>>=20 >>>=20 >>> FYI: >>>=20 >>> It is not your specific experiment, but I started my >>> "extra load" experimenst with . . . >>>=20 >>> I started a -j32 buildworld buildkernel with dnetc still >>> running. I'm generally seeing around 55% Active and 42% >>=20 >> Note "Active": user, sorry. >>=20 >>> nice, < 2% system (it was building libllvm at this point). >>> At that time: >>>=20 >>> load averages: 64.41, 60.52, 49.81 MaxObs: 64.47, 60.52, 49.81 >>>=20 >>=20 >> Contrasting results for some obj-lib32 build activity: >> much more variety of User, nice, and system, including >> times with < 5% user, 90+% nice. But not typical overall. >> But lots of time roughly around 50%/50% or 35%/60%. There >> were times with 15+% system. >>=20 >> Somewhat after buildkernel started: >>=20 >> load averages: 69.15, 64.12, 58.72 MaxObs: 75.98, 64.12, 58.72 >>=20 >> Harder to summarize, so overall timing reports from the >> buildworld and buildkernel stages. >>=20 >>=20 >> buildworld: >>=20 >> -------------------------------------------------------------- >> ... World build completed on Wed Mar 22 16:37:57 PDT 2023 >> ... World built in 2615 seconds, ncpu: 32, make -j32 >> -------------------------------------------------------------- >>=20 >>=20 >> buildkernel: >>=20 >> -------------------------------------------------------------- >> ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 16:43:10 = PDT 2023 >> -------------------------------------------------------------- >> ... Kernel(s) GENERIC-NODBG built in 311 seconds, ncpu: 32, make = -j32 >> -------------------------------------------------------------- >>=20 >> Afterwards: >>=20 >> load averages: 36.08, 53.14, 55.79 MaxObs: 75.98, 65.77, 59.84 >>=20 >>=20 >> I then did (not all in the same window): >>=20 >> $ sudo service dnetc onestop >> # rm -fr /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/ >>=20 >> before another -j32 buildworld buildkernel (no dnetc). The >> reuslts for this were: >>=20 >>=20 >> buildworld: >>=20 >> -------------------------------------------------------------- >> ... World build completed on Wed Mar 22 17:39:19 PDT 2023 >> ... World built in 1240 seconds, ncpu: 32, make -j32 >> -------------------------------------------------------------- >>=20 >> (compared to the 2615 for dnetc also in use) >>=20 >>=20 >> buildkernel: >>=20 >> -------------------------------------------------------------- >> ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 17:41:17 = PDT 2023 >> -------------------------------------------------------------- >> ... Kernel(s) GENERIC-NODBG built in 118 seconds, ncpu: 32, make = -j32 >> -------------------------------------------------------------- >>=20 >> (compared to the 311 for dnetc also in use) >=20 > I forgot to show the MaxObs load averages for the no-dnetc > context: >=20 > MaxObs: 39.77, 32.15, 25.75 >=20 >> Experiments without -j32 will take a lot longer, even >> without dnetc in use. I'm not sure there will be such >> results today. >>=20 >=20 I decided to do some more of the less time consuming testing. SCHED_4BSD, no dnetc, -j32 buildworld buildkernel : buildworld: -------------------------------------------------------------- ... World build completed on Wed Mar 22 19:16:35 PDT 2023 ... World built in 1235 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- (compared to 1240 for SCHED_ULE) So: no significant difference. buildkernel (SCHED_4BSD building a SCHED_4BSD): -------------------------------------------------------------- ... Kernel build for GENERIC-NODBG-SCHED_4BSD completed on Wed Mar 22 = 19:18:34 PDT 2023 -------------------------------------------------------------- ... Kernel(s) GENERIC-NODBG-SCHED_4BSD built in 119 seconds, ncpu: 32, = make -j32 -------------------------------------------------------------- (compared to 118 for SCHED_ULE building a SCHED_ULE) So: no significant difference. I'll try it with dnetc also active. =3D=3D=3D Mark Millard marklmi at yahoo.com