From nobody Mon Jul 19 00:37:43 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 248BC127E7F4 for ; Mon, 19 Jul 2021 00:37:47 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GSjcB4HFMz4YH7 for ; Mon, 19 Jul 2021 00:37:46 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by mail-wm1-x32c.google.com with SMTP id n4so9280614wms.1 for ; Sun, 18 Jul 2021 17:37:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=date:from:to:subject:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=3ESdynMmjo1CpDrX5uBKCZF7QKMuEGwVpXcEpZ7zgjw=; b=ardpPb7AAJAk/qLLymYWazHGiH7LJuvKHHlrA32/KEN4gmrhPlB6ttXEpBLc4aLDXf eATxXbxX39GpZKeEXtKnuBHZmaDrryI1+pM5zh0VTx2iTRgk71BPf7Sbdh1HYSywe5nU 2BCZh3hr2Hq8djdEBQsBrvl2zz+mY71B7tU6EHCINnoGk38SqLmNgehekm25qPILY6ne sm84T0nLQqq9dF6LIBibfS07BvL7lCOX1GBbaz6pYhKJG/rneX6d0MmpaAzXz9+OpBAu RfbE3ZpXWTYXNin1/ak14dIjart+QXxQhvKPYOWEFG0EwtFjSzxwnBlBli+6B0SCOxHh TuIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3ESdynMmjo1CpDrX5uBKCZF7QKMuEGwVpXcEpZ7zgjw=; b=grWtf3/Qqo4RFxzVWfYhV6McDYJ2Mi3XTEvlZm4NIvftpV0qketi+G5OFJJ2UQZ/48 LkQddIPqVL3lqyM0ERq8LcfiNg3Kb3xFJ3E/iuv6daJpcYFc1XV4ntEJy+uJe1bG9cRY 0LYf9vcxoAgo5JtkzCzNcOQaUhq0NJAy3slQO7GmrTxLwsgGlPEP0p+9Tyks1PpLjEp/ tkxgJOLjBSIDRjQPPRGU69q5XmSrIjeGJtUSX6I3RzpwawUi8FpWTCDKMsQJMega4h/y El2q/b9RxfqUXeQm1Vrx5T29LgUlUa+5tDvVlZYHt2k8qgVNLhI2F1i1t2V9hkw/7Fkq trmQ== X-Gm-Message-State: AOAM530xC8QGeQ9gK0pUGj5J4n6Y3m8p2jGMPgEkHlVQrXrnTUufLMUc 6uYt92t2XVOKXYQyc+c0nKl5ZtRRTsA= X-Google-Smtp-Source: ABdhPJyZfK357djpPR9zLGaBK9zrQ4z4ZEuEZt9jRX5OKkcUf9M2sCuVhOCEgmPgU1AuTRKyniYNDg== X-Received: by 2002:a7b:c955:: with SMTP id i21mr29140131wml.147.1626655065201; Sun, 18 Jul 2021 17:37:45 -0700 (PDT) Received: from gumby.homeunix.com (host-92-1-116-226.as13285.net. [92.1.116.226]) by smtp.gmail.com with ESMTPSA id d8sm19175260wrv.20.2021.07.18.17.37.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Jul 2021 17:37:44 -0700 (PDT) Date: Mon, 19 Jul 2021 01:37:43 +0100 To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: <20210719013743.0590b1f2@gumby.homeunix.com> In-Reply-To: <8239e474-fc36-b8aa-93b7-39197534cd30@heuristicsystems.com.au> References: <13445948-7804-20b4-4ae6-aaac14d11e87@m5p.com> <20210708101907.0be3a3c2@rimwks.local> <20210714164745.0128ea15@gumby.homeunix.com> <8239e474-fc36-b8aa-93b7-39197534cd30@heuristicsystems.com.au> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; amd64-portbld-freebsd12.2) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4GSjcB4HFMz4YH7 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=googlemail.com header.s=20161025 header.b=ardpPb7A; dmarc=pass (policy=quarantine) header.from=googlemail.com; spf=pass (mx1.freebsd.org: domain of rwmaillists@googlemail.com designates 2a00:1450:4864:20::32c as permitted sender) smtp.mailfrom=rwmaillists@googlemail.com X-Spamd-Result: default: False [0.72 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[googlemail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[googlemail.com:+]; DMARC_POLICY_ALLOW(-0.50)[googlemail.com,quarantine]; RECEIVED_SPAMHAUS_PBL(0.00)[92.1.116.226:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[googlemail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::32c:from]; DWL_DNSWL_NONE(0.00)[googlemail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[googlemail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.24)[-0.244]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::32c:from:127.0.2.255]; NEURAL_SPAM_SHORT(0.96)[0.962]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32c:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] Reply-To: rwmaillists@googlemail.com From: RW via freebsd-hackers X-Original-From: RW X-ThisMailContainsUnwantedMimeParts: N On Thu, 15 Jul 2021 11:03:04 +1000 Dewayne Geraghty wrote: > On 15/07/2021 1:47 am, RW via freebsd-hackers wrote: > > kern.sched.preempt_thresh=3D224 > > ... > > I think the default only allows preemption by real-time and kernel > > threads.=20 > > =20 > Hi RW,=C2=A0 Note the PRI(ority) column when you perform /usr/bin/top.=C2= =A0 > Processes with a PRI below the default kern.sched.preempt_thresh=3D80 > (ie nice -n 8) may pre-empt other processes or send interprocessor > interrupts to others (CPUs). I haven't got time to look into this in detail but from a cursory examination it looks like there must be some kind of translation between the PRI values seen in top and the priorities used in the scheduler. A threshold of 80 would be sensible in the context of top. I just ran a test and a cpu-bound process got a PRI of 85. But this seems to be a pure coincidence. kern.sched.preempt_thresh is being compared with the schedulers internal priorities and is defaulted to PRI_MIN_KERN, the highest priority in the kernel range, one level below realtime. =46rom sys/sys/priority.h #define PRI_MIN_REALTIME (48) #define PRI_MAX_REALTIME (PRI_MIN_KERN - 1) #define PRI_MIN_KERN (80) #define PRI_MAX_KERN (PRI_MIN_TIMESHARE - 1) #define PRI_MIN_TIMESHARE (120) #define PRI_MAX_TIMESHARE (PRI_MIN_IDLE - 1) #define PRI_MIN_IDLE (224) > idprio 0 top > is assigned a starting PRI of 124; so on SCHED_ULE, these processes > will receive cpu time (even at idprio 31) but won't pre-empt others. >=20 > If you really want all processes to pre-empt others, enabling > FULL_PREEMPTION achieves the same goal as 224.=C2=A0 I don't have a use > case for no pre-emption. Anyone? >=20 > Why kern.sched.preempt_thresh=3D224 helps desktop users, I can only > speculate that with a high threshold, more IPI's are sent to other CPU > cores so they can be busy (?).=C2=A0 Refer to > /usr/src/sys/kern/sched_ule.c -- > Returning to the topic.=C2=A0 Its a very hard choice between schedulers.= =C2=A0 I > did a lot of testing between them and tuning to see if one excelled on > my humble Xeon-E3.=C2=A0 I couldn't see a significant difference between > workloads - though next time (and a hint for others) I'll disable SMT > and set dev.cpu.0.freq to disable turbo behaviour.=C2=A0 For now, > sched_4bsd appears to be more efficient in terms of code complexity > and people with high CPU workloads have preferred sched_4bsd in the > past, while sched_ule has a lot of things to tweak and is recommended > by the FreeBSD project. Otherwise it wouldn't be the default=C2=A0=20 >=20 > Looking at > https://github.com/freebsd/freebsd-src/tree/main/sys/kern/sched_*.c=C2=A0 > their histories are tweaked a couple of times a year, so I wouldn't > rule sched_4bsd out of contention just yet.=20 >=20 > FWIW, my servers modify only: > kern.sched.affinity=3D7 > kern.sched.interact=3D0 > kern.sched.slice=3D128 > while firewalls: > kern.sched.balance=3D0 > kern.sched.interact=3D0 >=20 > A loadable schedule has been discussed here a few times - I vaguely > recall it being inefficient (complexity) and unnecessary (you'll > determine one scheduler and unless testing, unlikely to change).=C2=A0 > Further in the past, sched_4bsd was to be removed, but some > demonstrated it had better performance for their workload. > Cheerio. >=20 >=20 >=20 >=20