From nobody Fri Mar 24 19:47:08 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pjt5s25dlz412r9 for ; Fri, 24 Mar 2023 19:47:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-55.consmr.mail.gq1.yahoo.com (sonic308-55.consmr.mail.gq1.yahoo.com [98.137.68.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pjt5r0MT9z3H11 for ; Fri, 24 Mar 2023 19:47:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=SBV6cF3s; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679687246; bh=DwXwyxxskJyWupENkNVo0mByS+/KjOwwHwEkZzWuCwM=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=SBV6cF3sYY2XcV8WIA/P5RxPHvxVx8tmEq+yXjLLwtZP4dI38xqOJbm9VjPc0x9tprx556pfzVAvYAuvK9iRIrFcRNyBnKW30sSOnaUXyYHDf0r+BrtiT1JRZEYFK7uYF2Fsk+0iIujxemY+1Blqeh3abEWeCixPJkeXlr6+WRrYxENiJHMGdlSmc0asXv6jb5b9dWNPHJ8hSwqXtop+AchFW7MTc3pywOqPeKjiIETMz0UJKIITjtSga/Ydi1PXiVv7xJZlFxxCDIo1JgBW3yO2FP4/zOEs04Dl1DaaiYjjkASUvx3HbhHIMMx8s9TqIZ5SCdYOidSQXPnIRldL5w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679687246; bh=D+uMj/4+Om5O4gLKi1BJJKqOTHejYSu6Iricmaligcu=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Tkeou6Q9P0VBsWkGCv/lBAvKw/FRzfY+NWlJNttzdHbB/gMnuHQ41EE2Osyzp1qce7LHczO4RQB1O3KUQW3tycIlCPuHpRMqXUub1rCmfFgPp720RPO6aflzoYhbVRA89ozz8QhbQtozqXDmnQCcNYE+6gq6vquxk1a5SI2iNwlCD/cX66ghtvatljnmVOq7gy3WVqDe3K1y2oJ57/g6txd4Y6bAYMcn5KYzB1ilhyWxpFWaeFn0R1j39MVNkHcgu06ySQKn2yHER53xMxZ9xsr07RzlXLQ6LusBFPs4NfzClVYxemJ+xAekpHzFP5XD3KY9qd8L4FeZ4NFed9jhgQ== X-YMail-OSG: iwTPm5YVM1n9GL7nHc5yzzyOBE1OgU43sWDEnEKaQXhTwh9CGlga.4tmsI231T9 1PVrR7qT348Iwta0FqGImPHsFI9vdBAOCmIQrM.nBqPZ5c6NKfasdI4au_nXBoCcbHKaDmxHmdCG YZI_Hbdy4.evAOpEa1Cx4pRiKO27THhYM4d.yJ9aJnp44jChfZt.CxlOBP7o.HR4ksb97BCTLmjE J4eZSzhBbdN0NX0xCACuS9PsIyRrrZxTrcpcNLvY6tl3DvEjbUgx2Hgfy6ZngvH4ojEVL4rMSFkG o_q.sONfXD4IRcpjAq_H9c_6a6vd5Squ0yzIWpK0qZiq2iexebbusc5uVZYW2kwEnp0rXKG1oMcX WjKC6MBYMUXy5HqL8QOUB8Wfe3nY.p60yRyMQWPmuIZW0tpDpNyBMpylyPVpw4JKzvHXztROOKCx 857hSZOiPcpuxfX3MvcyyEUUzorV4M.9WmoOse6JcjRQUJxxWTZYSMrWmaQxuDguoYbmhcH5eBu1 ajs_Zv7K1_Y9fA.dvaTpLVPyEtt.xGMRgszvjGk7aPU8_XycqfPv6cKwVEXhEnmqMgJcZd4IsRFb 7yySCGlELFNMPKUa5TJfUUGhdt95pzwy6xGfvr7AWWkKs15C0hpcR44NMhZmjI.JAt8d0uPl6PUf 12Oh1s3mcU43V9gRI2bkgFiBO.Y5pZ1Z8GhP6Uc26HTiPMryKKxmRFVPZliSMc.e_544pNNbKeCY 6vrfMxGUeZgQRbSRJ26_vmwlJIb6BLyyCNsBLYLziG2UycVPX.wEyzytiw26afazDa.YDSuPkhZ6 2CaXmXHKN6i35NiyMvR5DNI96IwgTLek19ALN_nUtSTDVHGyX3fLQqlttcxWzJn89BCEljG53dGc Uc1WeW2qULrFyS9ON054JywzJG8Iial9_AAYfq5INcBCdGk3B5kP5_VMqvSoEi6GFbsrkgBKJ6Zw D8S77RX50QT.ZiXcAJJeKZTBSgpz0ogH_JJFdtsx81LYXvBtTWpIB099GXBjdxqL8Cw1adwPFQYs JxuMf2rkumHPblZeKqfYOrYAWnK2laFceAYeY.wbHguoSKJNbcUeTqzkt.dtApmemTkBQIWIpo7c O5tciWu6yQscPP7W1NhCBXCDJ4MUwcekib.s87P4VeU5aTSpGeKiF41s.0OctqXUeYhhJtyJuSvL TjKGAUeyCJXrDIeA0diGtYDj.1P_Auio4TA07CWGDGAOaI0Po4EgHjTZV3db79ctWW7D7EkPePca DAtPadmnQqXYxvRNmeBlCXix2M26at.d5PD4CsgHVKF72N5ST28FxndF_zspL94Dnht2899rum7f GQYy_rYzMZJgWN5QLSe2BkdwPDHo2bE_g4M_8_ApBYyrvI.V7XAJX41FHBUnxPpH6II.5.bxjr6H hDqdhAqDIgVvtHlNfooU0V0BRKI_DHVaGJ7lXK1xF.eMNkETb9Et0M42ciq1g_ANfwVsdGX46f8O JfEji3i443yhiBCttzjxfwGXVHks1ZkUlqz24JhlaYr5RWd3rM_zD2GiPk1TR69lYaXpgOcIKH1a dK3TrKv_JI1a49400QEUuSyTPdXcXb5RwtnlkpvRxpFkU22l87rnsu6WJXMgS_DQaiAb7x1Y8i_8 8BY_KddsNo8ljtArcc.7G9pTmQMrO7vUYd5uBBgvbnB07ZDsx1KYH3Hm3Rrug_keNggzYvRCaSha EqYmmrFGVnCgmlFSfOTy571zUTOkGK9uuo.K8zN3WXPMQbuQpyohfJIoTwq6e8_drXQjUvWyvzVJ s9_TTdGfy9bRssX4Zxw3tFa6UrLumUocFahgsxfD4DEHJSgCMJiS9RJN1d_BFXMyFUJmOrwHDjPT CBj9W4epFOhBPBeRwGfyIwoVU9QQqn.M9TuGiueIKdWNGLbmu.exWdL72NhUTzqv2h2rj.sOf.K0 nSWyZ1NYOEBPtGDNGL0MO0hvobdo6XyJXD8HXsL.Sra98QW7rOhNJ.CCD.ZfGDvA9QYS0uafrMJn feMfGHukepUbvTZ0ewQ_o3e4JVTai9uOnHFYA9FCwv51V5mh2QmbqWjNluncaUZ_ZSvu957lKvKC NTww73CYx8tZquJcG3SMPl2.Yr0bHLIk1itbGFA2BcG5fnZdlJpn.yXNz2vgW2AO9_P17G7xpD_9 pxymV9KrbrqPSA_idL359TDNcoBwgxe3dkDFOJGLMryA5K_b7mzrJhiTgBeT25Wzi369t_CLPP0a nxgUGOZ6N5iIa4rhwA8qvW0GIl6.UT5pmmbGwJOqYlUMyKOvbo.z6yPITsMRQ62S_QZzCdGf9pwL vQQ-- X-Sonic-MF: X-Sonic-ID: ac16413d-dc9f-4a39-9535-7d2ade4938b6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Fri, 24 Mar 2023 19:47:26 +0000 Received: by hermes--production-bf1-777648578f-lk5ld (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 5dd19e69c26d88815e55cd8d8a6d623e; Fri, 24 Mar 2023 19:47:20 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE Message-Id: <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732@yahoo.com> Date: Fri, 24 Mar 2023 12:47:08 -0700 To: Steve Kargl , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732.ref@yahoo.com> X-Spamd-Result: default: False [-3.39 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.89)[-0.888]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.31:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Pjt5r0MT9z3H11 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Steve Kargl wrote on Date: Wed, 22 Mar 2023 19:04:06 UTC : > On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: > >=20 > > Yes, there are reports that FreeBSD is not responsive by default - = but this > > may make it get overall better throughput at the expense of = responsiveness, > > because it might be doing fewer context switches. So just = complaining about > > a longer buildworld without seeing how much dnetc did in the same = wallclock > > time period is useless. Periodic rant's don't fix this lack of = information. > >=20 >=20 > I reported the issue with ULE some 15 to 20 years ago. > I gave up reporting the issue. The individuals with the > requisite skills to hack on ULE did not; and yes, I lack > those skills. The path of least resistance is to use > 4BSD. >=20 > % cat a.f90 > ! > ! Silly numerically intensive computation. > ! > program foo > implicit none > integer, parameter :: m =3D 200, n =3D 1000, dp =3D kind(1.d0) > integer i > real(dp) x > real(dp), allocatable :: a(:,:), b(:,:), c(:,:) > call random_init(.true., .true.) > allocate(a(n,n), b(n,n)) > do i =3D 1, m > call random_number(a) > call random_number(b) > c =3D matmul(a,b) > x =3D sum(c) > if (x < 0) stop 'Whoops' > end do > end program foo > % gfortran11 -o z -O3 -march=3Dnative a.f90=20 > % time ./z > 42.16 real 42.04 user 0.09 sys > % cat foo > #! /bin/csh > # > # Launch NCPU+1 images with a 1 second delay > # > foreach i (1 2 3 4 5 6 7 8 9) > ./z & > sleep 1 > end > % ./foo >=20 > In another xterm, you can watch the 9 images. >=20 > % top > st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 11:43:01 > 74 processes: 10 running, 64 sleeping > CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle > Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G = Free > Swap: 16G Total, 16G Free >=20 > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z > 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z > 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z > 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z > 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z > 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z > 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z > 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z > 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z >=20 > With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's = exit > after 55-ish seconds. If you try this experiment on ULE, you'll get = NCPU-1 > ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as = the > two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, > this was/is due to ULE's cpu affinity where processes never migrate to > another cpu. Admittedly, this was several years ago. Maybe ULE has > gotten better, but George's rant seems to suggest otherwise. Note: I'm only beginning to explore your report/case. There is a significant difference in your report and George's report: his is tied to nice use (and I've replicated there being SCHED_4BSD vs. SCHED_ULE consequences in the same direction George reports but with much larger process counts involved). In those types of experiments, I without the nice use I did not find notable differences. But it is a rather different context than your examples. Thus the below as a start on separate experiments closer to what you report using. Not (yet) having a Fortran set up I did some simple expriments with stress --cpu N (N processes looping sqrt calculations) and top. In top I sorted by pid to make it obvious if a fixed process was getting a fixed CPU or WCPU. (I tried looking at both CPU and WCPU, varying the time between samples as well. I also varied stress's --backoff N . This was on a ThreadRipper 1950X (32 hardware threads, so 16 cores) that was running: # uname -apKU FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #95 = main-n261544-cee09bda03c8-dirty: Wed Mar 15 19:44:40 PDT 2023 = root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG amd64 amd64 1400082 1400082 (That is a SCHED_ULE kernel. "GENERIC-NODBG-SCHED_4BSD" would be listed for SCHED_4BSD if I end up doing experiments with it.) Trying things like --cpu 33 I never got a process that sustained a low CPU or WCPU figure. The low figures moved around across the processes as I watched. When I tried the likes of --cpu 48 all the CPU or WCPU figures were generally lower than 99% but also showed variable figures that moved around across the processes. It definitely did not produce a sustained, approximately uniform figure across the processes. For comparison/contrast: --cpu 32 does have all the 32 processes showing 99+% all the time. This seems at least suggestive that, in my context, the specific old behavior that you report does not show up, at least on the timescales that I was observing at. It still might not be something you would find appropriate, but its does appear to at least be different. There is the possibility that stress --cpu N leads to more being involved than I expect and that such is contributing to the behavior that I've observed. =3D=3D=3D Mark Millard marklmi at yahoo.com