From nobody Mon Aug 10 20:42:29 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hJmsL1SrLz6jxkn for ; Mon, 10 Aug 2026 20:42:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hJmsL02Yhz3SwX for ; Mon, 10 Aug 2026 20:42:30 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1786394550; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=PQHx9nvdcbhyCx5B6Dw+awZTgoLf1HNej86KAI0Cojs=; b=Qaw69kG7ue3jydz6ZIIzleGtihTD7obEPMMyqZKJCp3v6ge3nkb0ivv/L75jut7ECkrIMp iPGS9Gs/86VDPPGTMopE5IJ2qh2JleWjfAdoSwj70XDoK3wHNbzTD3yPRMB5xmoka5H6Zf 0vbIBkSmDZN7gwYDkxg5S7mQH4Tcmcf9xqO0OStzljwQPgkNGYfRvATI13JGu66vY3o0pa lge2MrKzyMTeUP8/EMmS33uh/l6EJMfWv1CFneOpmohh8XmJ9XKrT9htkKmk3AOaxkRzSA etnKqPVc32YDC2M277Lh1xNV0UkIkVAkzb3LpckxmySc+NwQr2PjFBBSXxA5Wg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1786394550; a=rsa-sha256; cv=none; b=aLoKda+hFWfvMBlIrk1+MyC/o3toVADQm+Gwnxyc3+cOlmP4nXk1NUB1ZOHcu8NWkHWn8u 0O5mUHlo4MkEMKemOdgyRMejoQSR8M2CVdGN1Gcm4vwJvR/o48cqIaJvHzW89dvaDZzslJ ubTP4xsVp1+IfiEeSp2kBTbaX1HoLmvUhpxYOkjNkI3KOkUHwAA85WQReLz9FmaSUS1vfm rwcmL0dNJVh6Scl47KPEXFomisRlBijkiLO7SAjSF1X+IPC+08rYvlJjFQyPMnyupSgB92 GtzeunGRXARZX+gKI7fP4vl5Ng6SkZVemixZqAZhdCjPYdj+Sl84C93CQ33POw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1786394550; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=PQHx9nvdcbhyCx5B6Dw+awZTgoLf1HNej86KAI0Cojs=; b=pP4+cQDVCcvqMqn0nJzIvlVYvliOlNhgXg0GyT1cA+kNsIAIzBhoU/Zq6BVTRZuMx5S56h bUjiJUSw2kj46Vcumvn/B0RajdwHWmz+FZgrpFC86QYoGAGrybehHy9CmSL/MHlHqfrCpS s7oIwwFBTRagyK1qx0At9W8MVeNI+41080h5yGMYlqwOTSBtGyhienotACgY4lWlQYm8hq j7DYIFLrCsow5/mQAymXgjH/nUBYBrVu7PTJp7JFN3WpD8ts4VzoG3WtVb9YByh1v2MTa+ o404hFP6uBEbDJUIwANl8T+VFt9YmGUBN2vucy8gJKSSOlUxwaT/7Qgvnf2Eow== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hJmsK5wvrz92T for ; Mon, 10 Aug 2026 20:42:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 23e8a by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Mon, 10 Aug 2026 20:42:29 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alexander Motin Subject: git: a6847ba19592 - main - taskqueue: Move wakeup() out of tq_mutex List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: a6847ba19592c0ac60326e6e1b421cf6e344016b Auto-Submitted: auto-generated Date: Mon, 10 Aug 2026 20:42:29 +0000 Message-Id: <6a7a37b5.23e8a.11eb7d39@gitrepo.freebsd.org> The branch main has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=a6847ba19592c0ac60326e6e1b421cf6e344016b commit a6847ba19592c0ac60326e6e1b421cf6e344016b Author: Alexander Motin AuthorDate: 2026-08-10 20:18:34 +0000 Commit: Alexander Motin CommitDate: 2026-08-10 20:42:20 +0000 taskqueue: Move wakeup() out of tq_mutex taskqueue KPI require wakeup() to be called for each completed task. With everything else there heavily optimized over the years, even when doing nothing this wakeup()'s lock/unlock is significant. Since no external taskqueue consumer can depend on the tq_mutex, we can move the wakeup() out of it. It creates some complications for internal waiters, but those should be much more rare, and can be handled with separate locked wakeups on demand. My tests of taskqueue-intensive ZFS RAIDZ writes on 64-core system show performance improvement from this change ~4%, while same time reducing CPU usage by several percent due to lower lock contention, confirmed by CPU profiler. --- sys/kern/subr_taskqueue.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/sys/kern/subr_taskqueue.c b/sys/kern/subr_taskqueue.c index b337aa83d69d..f1f6f18bccdc 100644 --- a/sys/kern/subr_taskqueue.c +++ b/sys/kern/subr_taskqueue.c @@ -58,6 +58,7 @@ struct taskqueue_busy { struct task *tb_running; u_int tb_seq; bool tb_canceling; + bool tb_wanted; LIST_ENTRY(taskqueue_busy) tb_link; }; @@ -137,6 +138,15 @@ TQ_SLEEP(struct taskqueue *tq, void *p, const char *wm) return (msleep(p, &tq->tq_mutex, 0, wm, 0)); } +static __inline int +TQ_SLEEP_BUSY(struct taskqueue *tq, struct taskqueue_busy *tb, const char *wm) +{ + + TQ_ASSERT_LOCKED(tq); + tb->tb_wanted = true; + return (TQ_SLEEP(tq, tb, wm)); +} + static struct taskqueue_busy * task_get_busy(struct taskqueue *queue, struct task *task) { @@ -462,7 +472,7 @@ taskqueue_drain_tq_active(struct taskqueue *queue) restart: LIST_FOREACH(tb, &queue->tq_active, tb_link) { if ((int)(tb->tb_seq - seq) <= 0) { - TQ_SLEEP(queue, tb->tb_running, "tq_adrain"); + TQ_SLEEP_BUSY(queue, tb, "tq_adrain"); goto restart; } } @@ -506,6 +516,7 @@ taskqueue_run_locked(struct taskqueue *queue) KASSERT(queue != NULL, ("tq is NULL")); TQ_ASSERT_LOCKED(queue); tb.tb_running = NULL; + tb.tb_wanted = false; LIST_INSERT_HEAD(&queue->tq_active, &tb, tb_link); epochtasks = 0; @@ -534,8 +545,13 @@ taskqueue_run_locked(struct taskqueue *queue) epochtasks = 0; } - TQ_LOCK(queue); wakeup(task); + + TQ_LOCK(queue); + if (__predict_false(tb.tb_wanted)) { + tb.tb_wanted = false; + wakeup(&tb); + } } if (epochtasks > 0) NET_EPOCH_EXIT(et); @@ -628,13 +644,20 @@ taskqueue_cancel_timeout(struct taskqueue *queue, void taskqueue_drain(struct taskqueue *queue, struct task *task) { + struct taskqueue_busy *tb; if (!queue->tq_spin) WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK, NULL, __func__); TQ_LOCK(queue); - while (task->ta_pending != 0 || task_get_busy(queue, task) != NULL) - TQ_SLEEP(queue, task, "tq_drain"); + for (;;) { + if (task->ta_pending != 0) + TQ_SLEEP(queue, task, "tq_drain"); + else if ((tb = task_get_busy(queue, task)) != NULL) + TQ_SLEEP_BUSY(queue, tb, "tq_drain"); + else + break; + } TQ_UNLOCK(queue); }