From nobody Thu Aug 20 17:53:58 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hQrfH1L7Vz6pFRx for ; Thu, 20 Aug 2026 17:53:59 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hQrfG4fxnz41QX for ; Thu, 20 Aug 2026 17:53:58 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787248438; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fnL72pU5MOy8XJBzGd8+ftaWvYyfTQuxxQDLgFw61kw=; b=B9DmcSRY3O6SR/LU+QnARunD744M1FkxboamIZ2IwHd/dokPIGJnWffhGCW+ONjVwXf6Dq eO3pDLM95Jxx6aRLZ2Lvm9YidF819f/OHO2YTdQRS3vmxYBBp+FcC3v/LViI0pl+D53Jxr 1txmV7JH69lN2xRqFiFyemmXDtCA3sJtv98Gh7twGLbbFeF2spS7oyX4zZq0aOPVadw/wC qPEmH8Chd6BUC0SeK1ifo040t/MeLUTJHI7s61pbhAU1PsmhRg6hxVbA13U2Nfc9MiZEIt mn81vdbN6e2suMt6g+N96G8y6x6Nj/DxVhTztGR0CYcbGfEVYffEKHF/4RPOjg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1787248438; a=rsa-sha256; cv=none; b=v0SXZXwJWpI6x/4Om5NNOHBOfehpEE3McOBGSdv0v9IWWrJ14vfOvDaXWO3GnaLGszVRRZ XvOs/hhZtb+w+jKn/1Suk/N3D1nfgmfo5hjaY53XsDlcrqOGZlemduCTw3Z4kr4z0sfcH8 ETkgKJ5UFQrlo14yVnsNPaJ7zKnwylR/OYtU+P6ve6ecApgec0BapOTLTw/8i38kuP+1Rm CaNBXG+pxkka4kPxQa1k+Oqnm31nosvd1xJGQQ+H7fMQOxtIQiwJEnguMY8nH7w26FTXEM shh1nyiurcyqnpZaDAkBlrgmK8ksgYLAy92LpoSVIp6TaCVUcSXXa/Ed0miyig== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1787248438; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fnL72pU5MOy8XJBzGd8+ftaWvYyfTQuxxQDLgFw61kw=; b=jDbW90n9ZXtDMdi2jH5GQ8wDDJAg3QuZqyAjIui6YATYcWGcxvDaR1VOL98fp26oCeOkJM EYJ8SeGgKCOX+IZttoNy5+ek69ezRxv3eErl/3oKr3S+dLvVWCAjZ4kpfiJ6uuJZKgOg76 fJD3r1ckxL0FNFYM8wWPlTz1KmVGHORT0vOxCqXDrSp5fEXJWF1CvR7YfQ6v6jwRTPTwBT kIBwg3rLFS+64kx+d63bZmo1XKZU4GJaQQnMkl7IYzAXOAqn0ymIGyeFesqor/x8Q0XIcb Ne9uwy1lDw1oJVaFbMHHqMUvsENiK2XxJ84k1NthJD5DK9hDuFINmaFJEcXhag== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hQrfG3ZvczBwc for ; Thu, 20 Aug 2026 17:53:58 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 475c0 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Thu, 20 Aug 2026 17:53:58 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org Cc: Mark Johnston From: Colin Percival Subject: git: bdfae0fed88f - releng/14.5 - epoch: Fix epoch_drain_callbacks() List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: cperciva X-Git-Repository: src X-Git-Refname: refs/heads/releng/14.5 X-Git-Reftype: branch X-Git-Commit: bdfae0fed88ffb32f5686399bad9ec4bd443daf6 Auto-Submitted: auto-generated Date: Thu, 20 Aug 2026 17:53:58 +0000 Message-Id: <6a873f36.475c0.5b45f50@gitrepo.freebsd.org> The branch releng/14.5 has been updated by cperciva: URL: https://cgit.FreeBSD.org/src/commit/?id=bdfae0fed88ffb32f5686399bad9ec4bd443daf6 commit bdfae0fed88ffb32f5686399bad9ec4bd443daf6 Author: Mark Johnston AuthorDate: 2026-07-08 17:13:01 +0000 Commit: Colin Percival CommitDate: 2026-08-20 17:53:07 +0000 epoch: Fix epoch_drain_callbacks() This function is supposed to wait until all pending callbacks have been executed. This is useful in some contexts where we tear down some context (like a VNET jail and its associated UMA zones) synchronously, and we want to make sure that all pending asynchronous callbacks (which may free objects to said UMA zones) have run first. The implementation schedules a callback on each CPU and waits for them all to run. This assumes that, on a given CPU, callbacks are executed in the order that they are pushed. This assumption depends on the implementation of epoch_call_task() and ck_epoch_poll_deferred(), and it is not true in general. Callbacks are pushed onto a per-CPU stack in LIFO order. ck_epoch_poll_deferred() first pulls out the callbacks from epoch - 2, which are always safe to execute, and in so doing reorders them such that the oldest callback as at the top of the stack, so in this case, epoch_call_task() will execute them in order. However, ck_epoch_poll_deferred() may determine that it is safe to execute callbacks from epoch - 1 (or even from the current epoch if there are no active readers), and in this case it will push those callbacks onto the returned stack. This means that epoch_call_task() will invoke those newer destructors before the older ones, which means that epoch_drain_callbacks() may return early. Fix the correctness problem by simply doing all of this twice: once the first callback is invoked, we know that all of the callbacks that were pending at the time that epoch_drain_callbacks() was called are scheduled to be executed, so when the second callback is executed we know that they must be finished. This is slow, but it is already slow, and the slowness is less noticeable after commit dce56594991. I note that in an ideal world, this function would not exist, and all of the teardown would happen asynchronously, rather than the current mismash of synchronous and asynchronous cleanup. Approved by: re (cperciva) PR: 290201 Reviewed by: glebius MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D58030 (cherry picked from commit 7bf11a2f0c9ad7af00996105fd34e61f1040402b) (cherry picked from commit 8eef4664327fcc2fe04eb74e03e2e793df90d5af) --- sys/kern/subr_epoch.c | 50 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/sys/kern/subr_epoch.c b/sys/kern/subr_epoch.c index fb7953c54a63..84ccce9f77b2 100644 --- a/sys/kern/subr_epoch.c +++ b/sys/kern/subr_epoch.c @@ -987,30 +987,40 @@ epoch_drain_callbacks(epoch_t epoch) sched_unbind(td); td->td_pinned = 0; - CPU_FOREACH(cpu) - epoch->e_drain_count++; - CPU_FOREACH(cpu) { - er = zpcpu_get_cpu(epoch->e_pcpu_record, cpu); - sched_bind(td, cpu); - epoch_call(epoch, &epoch_drain_cb, &er->er_drain_ctx); - } + /* + * Schedule a destructor on each CPU and block until all of them have + * run. Don't assume that destructors are invoked in order: once we're + * finished draining, do the same thing again to ensure that any + * destructors scheduled after the first pass have also run. + */ + for (int i = 0; i < 2; i++) { + CPU_FOREACH(cpu) + epoch->e_drain_count++; + CPU_FOREACH(cpu) { + er = zpcpu_get_cpu(epoch->e_pcpu_record, cpu); + sched_bind(td, cpu); + epoch_call(epoch, &epoch_drain_cb, &er->er_drain_ctx); + } - /* restore CPU binding, if any */ - if (was_bound != 0) { - sched_bind(td, old_cpu); - } else { - /* get thread back to initial CPU, if any */ - if (old_pinned != 0) + /* restore CPU binding, if any */ + if (was_bound != 0) { sched_bind(td, old_cpu); - sched_unbind(td); - } - /* restore pinned after bind */ - td->td_pinned = old_pinned; + } else { + /* get thread back to initial CPU, if any */ + if (old_pinned != 0) + sched_bind(td, old_cpu); + sched_unbind(td); + } + /* restore pinned after bind */ + td->td_pinned = old_pinned; + thread_unlock(td); - thread_unlock(td); + while (epoch->e_drain_count != 0) + msleep(epoch, &epoch->e_drain_mtx, PZERO, "EDRAIN", 0); - while (epoch->e_drain_count != 0) - msleep(epoch, &epoch->e_drain_mtx, PZERO, "EDRAIN", 0); + thread_lock(td); + } + thread_unlock(td); mtx_unlock(&epoch->e_drain_mtx); sx_xunlock(&epoch->e_drain_sx);