From nobody Mon Sep 07 08:42:40 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hdgYy17c2z6rj1j for ; Mon, 07 Sep 2026 08:42:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hdgYx6n2Zz448R for ; Mon, 07 Sep 2026 08:42:45 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1788770566; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=eEUX4IIRREYHHt7YtMdcd/0Fj01G5mYNP4NSs/746/Y=; b=xPh3Ioe37a1281eIey9NO23DEfWJxienJ3pdCH8b1fBnolLSUI8RgmrTvQjTCfdcrc4XTu F3rUb2ga5rNkxVYDq/WN7C7QlUYCXGZlTYBE/CkRUNyLH4D583z/0YMG7+PtEeVFqzrV6b A6GknyCPUwMHY+d8pKdTCXWuSPie3V1uHpyOnO5xWXoXXfaPwr2E1jrq2hsMkH0Po1lO3W enx+n3a2Y1kU2WLr8o8dj9KqP10qLp7chsbRvN3PByqMcHFRhxvwkwxHJRtFTnydD2g4M7 IvsOzo1i11jIyKmFxMTiiNfVy4N2FhC8gI4yiSW4o8DWfJmdz22CHExtyUBiUg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1788770566; a=rsa-sha256; cv=none; b=FVnRoMQXAihgVhYq3SsL19JdcMOSVBrn2gqn6bpBngJ0Rbp53k54wF2eR0ghjz8xce421T vjDdWx1QYreL5GQ1djsWjTl3kk5HAO09YE/4yhys36QAZc/ASE9FAVyGSca07qLCyUJMbv aI+dQIFrvlvm40j8VfmmnDFjCDNvLY6cvQ2bacF3NwmUuEz5k8wPzHmxRVJsF6iLqqOseD rYwxNGrFjxJhGNRTEKDRn95F2mPwPHmTnEqNCYje2udk1EFRMwkj4gRUfa5We+TFjrHoBf gZLZxVmaPhDWmI4Q1vHFAjcRY7QKU7IGHkrSGr4hed26Ie3bSv0rYmk3k5gwUg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1788770566; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=eEUX4IIRREYHHt7YtMdcd/0Fj01G5mYNP4NSs/746/Y=; b=ocMCEfk1XqYg9dYrA5qLeemhE7zIqFPXfQ46MYcVM1pa4aTBTlAFc2FQaNawSSThZzP18U nWhH/6Be1y3GuY6t/hYtb1uwnMZx9FzqsOow8ijJifZykEvbxpYKpbZf7Tn7jZVAbGIG2i t2F2ormIii813H6kxWH4yOSH0xNW4oh+MjNo7UrXSEmKb56xHnkKzvJYUOWOVs5nQBJA3k SobS/JvN96FKv0IGxaJs4s0FkrJCFRpFeevDNINsCc4yBaGtMxoaIKRU16j/JymRbIIJPs IBFt+yNxlKfQyPba4bCVwg0FXSVenfSwm0gZFcY7Vb84Wc2DkZCMvCiyEylngA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hdgYx2c4rz19JB for ; Mon, 07 Sep 2026 08:42:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3fc73 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Mon, 07 Sep 2026 08:42:40 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Olivier Certner Subject: git: afd1ae57702c - stable/15 - sched_ule: Fix selecting lowest priority thread early in corner case List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: olce X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: afd1ae57702c90990a4ac4873c320e7dde735809 Auto-Submitted: auto-generated Date: Mon, 07 Sep 2026 08:42:40 +0000 Message-Id: <6a9e7900.3fc73.105a89bb@gitrepo.freebsd.org> The branch stable/15 has been updated by olce: URL: https://cgit.FreeBSD.org/src/commit/?id=afd1ae57702c90990a4ac4873c320e7dde735809 commit afd1ae57702c90990a4ac4873c320e7dde735809 Author: Olivier Certner AuthorDate: 2026-06-16 21:06:36 +0000 Commit: Olivier Certner CommitDate: 2026-09-07 08:41:02 +0000 sched_ule: Fix selecting lowest priority thread early in corner case When transferring a thread with near 100% CPU statistics (but not 100%; up to 57.5/59≈97.46%) to a CPU where the enqueue offset is ahead of at least 2 from the dequeue one, which requires peculiar conditions to happen (transfer triggered by a bind request or cpuset change, or during balancing if a thread or more existed from a brief amount of time on the origin CPU), the transferred thread can get placed after the dequeue offset, effectively making it appear as a high priority one unduly, causing latency increase for other threads. The change here was missed when changing the enqueue and dequeue offsets update mechanism to recover pre-256-queue-runqueue ULE anti-starvation and fairness behavior. That change opened up the possibility that these two offsets are apart by more than one. Reviewed by: markj Discussed with: Minsoo Choo Fixes: 6792f3411f6d ("sched_ule: Recover previous nice and anti-starvation behaviors") MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D57829 (cherry picked from commit a6587ae0f46dba4e47fa92c831f55288ec181734) --- sys/kern/sched_ule.c | 45 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/sys/kern/sched_ule.c b/sys/kern/sched_ule.c index cf0baabfda86..a9fe49b11ff7 100644 --- a/sys/kern/sched_ule.c +++ b/sys/kern/sched_ule.c @@ -484,6 +484,17 @@ sched_shouldpreempt(int pri, int cpri, int remote) return (0); } +static inline int +normalize_ts_off(int offset) +{ + /* + * Adding RQ_TS_POL_MODULO before taking the modulo is to ensure the + * dividend is positive (we want a positive result). + */ + MPASS(offset >= -RQ_TS_POL_MODULO); + return ((offset + RQ_TS_POL_MODULO) % RQ_TS_POL_MODULO); +} + /* * Add a thread to the actual run-queue. Keeps transferable counts up to * date with what is actually on the run-queue. Selects the correct @@ -517,18 +528,34 @@ tdq_runq_add(struct tdq *tdq, struct thread *td, int flags) /* Current queue from which processes are being run. */ idx = tdq->tdq_ts_deq_off; else { - idx = (RQ_PRI_TO_QUEUE_IDX(pri) - RQ_TS_POL_MIN + - tdq->tdq_ts_off) % RQ_TS_POL_MODULO; + idx = normalize_ts_off( + /* Offset corresponding to priority. */ + RQ_PRI_TO_QUEUE_IDX(pri) - RQ_TS_POL_MIN + + /* Insertion offset. */ + tdq->tdq_ts_off); /* - * We avoid enqueuing low priority threads in the queue - * that we are still draining, effectively shortening - * the runqueue by one queue. + * We avoid enqueuing low priority threads in the queues + * we still have to drain. This effectively shortens + * the runqueue by a few queues (see update of + * 'tdq_ts_deq_off' in sched_clock()). + * + * The expressions that include differences below are + * measuring the "distance" from the dequeue offset to + * either 'idx' or the insertion offset modulo + * RQ_TS_POL_MODULO. Thanks to the arithmetic operators + * always performing the usual arithmetic conversions, + * all operands are promoted to integers, which is + * necessary to accomodate corner cases (else + * we would have to be conditional on whether the first + * term is greater or lower than the second, in the + * second case correcting the result with UCHAR_MAX % + * RQ_TS_POL_MODULO). */ if (tdq->tdq_ts_deq_off != tdq->tdq_ts_off && - idx == tdq->tdq_ts_deq_off) - /* Ensure the dividend is positive. */ - idx = (idx - 1 + RQ_TS_POL_MODULO) % - RQ_TS_POL_MODULO; + normalize_ts_off(idx - tdq->tdq_ts_deq_off) < + normalize_ts_off(tdq->tdq_ts_off - + tdq->tdq_ts_deq_off)) + idx = normalize_ts_off(tdq->tdq_ts_deq_off - 1); } /* Absolute queue index. */ idx += RQ_TS_POL_MIN;