From nobody Wed Mar 22 19:35:33 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phdx45lrlz41CP6 for ; Wed, 22 Mar 2023 19:35:36 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x35.google.com (mail-oa1-x35.google.com [IPv6:2001:4860:4864:20::35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phdx33Dt9z4D9s; Wed, 22 Mar 2023 19:35:35 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=kREsM9pW; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2001:4860:4864:20::35 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-x35.google.com with SMTP id 586e51a60fabf-17aaa51a911so20511243fac.5; Wed, 22 Mar 2023 12:35:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679513734; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=DNqC16//EEe720Gn7k2KbqRc3GnXMEeEFesuBSbgXOk=; b=kREsM9pWRRLSWhyxUQ2t8mHWUvXoRiWdQmXgpzCV/+Bpel8Gb619jmmmfkUtZ6vcQT KS83w3mxlLbDqOe5QcSR8WPSALg3+gT/Xd5AYlQcbEbSjq2LKAAaGQ/9k5Dqz1Sh5wOA 7MMBbuBve+eb1Eh547t523kb+/N93vexUEgAamVfGPGvIRCmKTMB+xZADafcP/sZcgCm Rxq6oKqS7XggNLmEn2Cd/DLzQjQUWX/AlshBvfeP6mGw5ezWozAKDnjEWhU462lQCW3x bXyPt98tB6DYV9EfMFuUMdOUGy9ijJrp5xj+sDtDswoQTqCRMc7TiZ6kn9Rd03sgdewN rmyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679513734; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=DNqC16//EEe720Gn7k2KbqRc3GnXMEeEFesuBSbgXOk=; b=JSoV3v/0rmpjtI+Z0EVm7j/RZd/C5xkeS07cZ2KGnqJqdKY0V4BvXeD18sMfEYkNzh tYIAVVpvtlA7gNQgVfB7l7lWFb5OUr6RCi/JfXIPy5rY9X1PAqf4Gka6mZejPbh/9g46 uHeZIt85D6TQBOyQOZVnUI/gOnimzHt3sQJD0bNQN4usvMW4KkzlEbpE/BEANr15MF1/ QVecsXIFC6e1X6TgSUyGiYWekliZySUt2vY+P8qId8gF9fFXEU3jiRDN6RUy2nWDY8eh 7TeQkzwb8X8jU2qeqaxEeNcGBxeGYjvMx8FYCdGNQOwP+fF8dSmbzcny11H2CiYpRQVW UWbg== X-Gm-Message-State: AO0yUKWKx4LAxgMbm6CS4GE45mRRgQCaNaYaCqh/PA8wLRcmzltq7QRR hjalVk7OVgEKUfgJ1+5fVMcBx3oFfJnmKFnCvrBj2f/Z X-Google-Smtp-Source: AKy350Y+1xkiqhmFjZCeu8d8zO22BnZmZuMlKxM/7Qnwb29Rss3mw9wwkM6NSwSYDQT6r9EOBbNYaielbK1ZWS8dL5A= X-Received: by 2002:a05:6871:d96:b0:17a:b713:63e9 with SMTP id vi22-20020a0568710d9600b0017ab71363e9mr330405oab.4.1679513734071; Wed, 22 Mar 2023 12:35:34 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1922:0:b0:49c:b071:b1e3 with HTTP; Wed, 22 Mar 2023 12:35:33 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Wed, 22 Mar 2023 20:35:33 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: sgk@troutmask.apl.washington.edu Cc: Matthias Andree , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.92 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; NEURAL_HAM_SHORT(-0.92)[-0.920]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::35:from]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Phdx33Dt9z4D9s X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/22/23, Mateusz Guzik wrote: > On 3/22/23, Steve Kargl wrote: >> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >>> >>> Yes, there are reports that FreeBSD is not responsive by default - but >>> this >>> may make it get overall better throughput at the expense of >>> responsiveness, >>> because it might be doing fewer context switches. So just complaining >>> about >>> a longer buildworld without seeing how much dnetc did in the same >>> wallclock >>> time period is useless. Periodic rant's don't fix this lack of >>> information. >>> >> >> I reported the issue with ULE some 15 to 20 years ago. >> I gave up reporting the issue. The individuals with the >> requisite skills to hack on ULE did not; and yes, I lack >> those skills. The path of least resistance is to use >> 4BSD. >> >> % cat a.f90 >> ! >> ! Silly numerically intensive computation. >> ! >> program foo >> implicit none >> integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) >> integer i >> real(dp) x >> real(dp), allocatable :: a(:,:), b(:,:), c(:,:) >> call random_init(.true., .true.) >> allocate(a(n,n), b(n,n)) >> do i = 1, m >> call random_number(a) >> call random_number(b) >> c = matmul(a,b) >> x = sum(c) >> if (x < 0) stop 'Whoops' >> end do >> end program foo >> % gfortran11 -o z -O3 -march=native a.f90 >> % time ./z >> 42.16 real 42.04 user 0.09 sys >> % cat foo >> #! /bin/csh >> # >> # Launch NCPU+1 images with a 1 second delay >> # >> foreach i (1 2 3 4 5 6 7 8 9) >> ./z & >> sleep 1 >> end >> % ./foo >> >> In another xterm, you can watch the 9 images. >> >> % top >> st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 >> 11:43:01 >> 74 processes: 10 running, 64 sleeping >> CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle >> Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G >> Free >> Swap: 16G Total, 16G Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU >> COMMAND >> 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z >> 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z >> 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z >> 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z >> 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z >> 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z >> 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z >> 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z >> 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z >> >> With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's exit >> after 55-ish seconds. If you try this experiment on ULE, you'll get >> NCPU-1 >> ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as the >> two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, >> this was/is due to ULE's cpu affinity where processes never migrate to >> another cpu. Admittedly, this was several years ago. Maybe ULE has >> gotten better, but George's rant seems to suggest otherwise. >> > > While I have not tried openmpi yet, I can confirm there is a problem > here, but the situtation is not as clear cut as one might think. > > sched_4bsd *round robins* all workers across all CPUs, which comes at > a performance *hit* compared to ule if number of workers is <= CPU > count -- here ule maintains affinity, avoiding cache busting. If you > slap in $cpu_count + 1 workers, 4bsd keeps the round robin equally > penalizing everyone, while ule mostly penalizes a select victim. By > the end of it, you get lower total cpu time spent, but higher total > real time. See below for an example. > > Two massive problems with 4bsd, apart from mandatory round robin which > also happens to help by accident: > 1. it has one *global* lock, meaning the scheduler itself just does > not scale and this is visible at modest contemporary scales > 2. it does not understand topology -- no accounting done for ht nor > numa, but i concede the latter wont be a factor for most people > > That said, ULE definitely has performance bugs which need to be fixed. > At least for the case below, 4BSD just "lucked" into sucking less > simply because it is so primitive. > > I wrote a cpu burning program (memset 1 MB in a loop, with enough > iterations to take ~20 seconds on its own). > > I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 distinct > cores. > > The test runs *9* workers, here is a sample run: > > 4bsd: > 23.18 real 20.81 user 0.00 sys > 23.26 real 20.81 user 0.00 sys > 23.30 real 20.81 user 0.00 sys > 23.34 real 20.82 user 0.00 sys > 23.41 real 20.81 user 0.00 sys > 23.41 real 20.80 user 0.00 sys > 23.42 real 20.80 user 0.00 sys > 23.53 real 20.81 user 0.00 sys > 23.60 real 20.80 user 0.00 sys > 187.31s user 0.02s system 793% cpu 23.606 total > > ule: > 20.67 real 20.04 user 0.00 sys > 20.97 real 20.00 user 0.00 sys > 21.45 real 20.29 user 0.00 sys > 21.51 real 20.22 user 0.00 sys > 22.77 real 20.04 user 0.00 sys > 22.78 real 20.26 user 0.00 sys > 23.42 real 20.04 user 0.00 sys > 24.07 real 20.30 user 0.00 sys > 24.46 real 20.16 user 0.00 sys > 181.41s user 0.07s system 741% cpu 24.465 total > > It reliably uses 187s user time on 4BSD and 181s on ULE. At the same > time it also reliably has massive time imblance between > fastest/slowest in terms of total real time between workers *and* ULE > reliably uses more total real time to finish the entire thing. > > iow this is a tradeoff, but most likely a bad one > In case I did not make it clear enough: 1. both schedulers have problems 2. ... 4bsd has problems which are infeasible to fix 3. ... ule has problems which may or may not be easy, remains to be seen, but i can't promise i will have time to dig into it -- Mateusz Guzik