From nobody Thu Mar 23 20:53:25 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjHcf25jcz40JrC for ; Thu, 23 Mar 2023 20:53:38 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjHcd5qsMz469g for ; Thu, 23 Mar 2023 20:53:37 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x534.google.com with SMTP id h8so212064ede.8 for ; Thu, 23 Mar 2023 13:53:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; t=1679604816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=1kmZuGnwDa/Bih5tEQG+siOUPEWXQHkR8OiWKbAoZ5s=; b=I5rMDDJwi5Ik/S2pW+9wYVHp0VgnGHtmRr1oSM5sTXWXdV5RjpQ3HOVv3WBj6DvjT1 R0gB4/Ylbsmg+xylwQ96nqy2u3S8H6DApZltMgudrEQbEKPN+Z61V5x3003ylCw4ejXZ tC51UUe4/46UMbaay3AxWk9gaGlF6OEDfG8cqKzjpy/B4KhPKiTaTq3OTHr1+akiUx+d npiIJ9QfQy+t0DY+PwInBQT/HP38JwMQ/qUM7DxfGzQMPdw+qY9f8RczXpxigg9PnbLo sM9Dbd2pS3EBRQvUIuYfbXdP6m7gYn6tDwi9Xe9ARkYQzQPw/gsqgnpEVVOC07fzqF0j TRPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679604816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1kmZuGnwDa/Bih5tEQG+siOUPEWXQHkR8OiWKbAoZ5s=; b=EU3P1qRBVj+ly3yKJk/WtAP5VvJLGM0PeRXp/eJmR6Q1HvA0i+SyTVs499m1LFyLD0 QxoJxLkRS8llHFWc9ODMp7Ps1AIqxn71b9XDDV3/ZFTS77N2lHUIVNzQq8xzqlOAekke Bt/4Uh4KEUeEhad+xHOD33hkWgasGnQmqxucoJn+wi2jeboP8T8iQZjRdPvjrgk5z2ep HSTQ9Z3Z8Zig3ri6zCba7amjsaDVW4dGchGAnz48gkePTkl9T8gDk3bwwaBDBcdSeWKZ 3zI0MT3BB/CRJKDZA3iiaZlNxe2YJrl5FGhuQqqMx+j1gYMqCf9ASjYagTT4kGBlvHz6 KXUg== X-Gm-Message-State: AAQBX9eBFhtTxNZRnQbU31MiFG2HTddK0kmyF3Q8qJ2secj8netg+JhQ GL91i6yFRUzb0QaOFZLhJJR8js06UldGhGofTs2lag== X-Google-Smtp-Source: AKy350a52m77qhUHhFElRXVqZHCyO/2ZDIdhbBV7Drj4t5IkIfOAQtRNqVNvHoW8ocxYp+IupAKq58BqzorCxBIz4X4= X-Received: by 2002:a17:906:5fca:b0:930:310:abcf with SMTP id k10-20020a1709065fca00b009300310abcfmr207497ejv.2.1679604816121; Thu, 23 Mar 2023 13:53:36 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39.ref@yahoo.com> <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39@yahoo.com> In-Reply-To: <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39@yahoo.com> From: Warner Losh Date: Thu, 23 Mar 2023 21:53:25 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Mark Millard Cc: FreeBSD Hackers Content-Type: multipart/alternative; boundary="0000000000006da3d505f79778ca" X-Rspamd-Queue-Id: 4PjHcd5qsMz469g X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000006da3d505f79778ca Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Mar 23, 2023, 9:46 PM Mark Millard wrote: > Warner Losh wrote on > Date: Wed, 22 Mar 2023 22:57:08 UTC : > > > On Wed, Mar 22, 2023 at 1:41=E2=80=AFPM George Mitchell > > wrote: > > > > > service dnetc start > > > I am literally running "make buildworld" with no additional options. > > > > > > > > So what are the results for make buildworld -j $(sysctl -n hw.ncpu )? > > > Note: My experiments have been in this -j $(sysctl -n hw.ncpu ) > realm. > > > ULE scales much better, but when there's too little to do it can make > poor > > choices. > > > > ULE is better locked and don't fall over on high core count systems lik= e > > BSD does at moderate load. > > (I'm presuming the above is not about the specifics > of the effectively different interpretations of the > likes of having extra "nice 20" activity by the two > schedulers for the examples related to the original > "rant", other than the -jN issue.) > > Any idea on what scale "high core count systems" need to be > for what sort of "moderate load" to end up with signficant > differences? Sched_bsd is basically unusable on my 64 core 128 thread machine with make -j 150 (nice or no). With ULE I don't notice. That's not to say ule can't be better (me not noticing is hardly scientific), but I tried sched bsd when I got the thread ripper and found the machine too unresponsive when I was doing large builds... But I wasn't playing video on this box... so maybe I hit a local optimal point... Warner What sort of context(s) show ULE scaling much > better? On the 16 core ThreadRipper 1950X (32 hardware > threads) I've really only demonstrated the "nice 20" > distinction as significant between the schedulers so far. > (I do not have acccess to anything with more hardware threads.) > > Note: I've not (yet?) been looking at having just a little > more than the number of hardware threads active (no nice > involvement). > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --0000000000006da3d505f79778ca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Mar 23, 2023, 9:46 PM Mark Millard <marklmi@yahoo.com> wrote:
Warner Losh <imp_at_bsdimp.com= > wrote on
Date: Wed, 22 Mar 2023 22:57:08 UTC :

> On Wed, Mar 22, 2023 at 1:41=E2=80=AFPM George Mitchell <geor= ge+freebsd@m5p.com>
> wrote:
>
> > service dnetc start
> > I am literally running "make buildworld" with no additi= onal options.
> >
> >
> So what are the results for make buildworld -j $(sysctl -n hw.ncpu )?<= br>

Note: My experiments have been in this -j $(sysctl -n hw.ncpu )
realm.

> ULE scales much better, but when there's too little to do it can m= ake poor
> choices.
>
> ULE is better locked and don't fall over on high core count system= s like
> BSD does at moderate load.

(I'm presuming the above is not about the specifics
of the effectively different interpretations of the
likes of having extra "nice 20" activity by the two
schedulers for the examples related to the original
"rant", other than the -jN issue.)

Any idea on what scale "high core count systems" need to be
for what sort of "moderate load" to end up with signficant
differences?

Sched_bsd is basically unusable on my 64 core 128 thread machine = with make -j 150 (nice or no). With ULE I don't notice. That's not = to say ule can't be better (me not noticing is hardly scientific), but = I tried sched bsd when I got the thread ripper and found the machine too un= responsive when I was doing large builds...=C2=A0
But I wasn't playing video on this box... so = maybe I hit a local optimal point...

Warner

What sort of context= (s) show ULE scaling much
better? On the 16 core ThreadRipper 1950X (32 hardware
threads) I've really only demonstrated the "nice 20"
distinction as significant between the schedulers so far.
(I do not have acccess to anything with more hardware threads.)

Note: I've not (yet?) been looking at having just a little
more than the number of hardware threads active (no nice
involvement).

=3D=3D=3D
Mark Millard
marklmi at yahoo.com

--0000000000006da3d505f79778ca--