git: 1744df209f7f - stable/15 - critical_exit_preempt(): Re-check for preemption locked
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 19 Nov 2026 15:32:14 UTC
The branch stable/15 has been updated by olce:
URL: https://cgit.FreeBSD.org/src/commit/?id=1744df209f7f6afa353574a2b91da74faefa9831
commit 1744df209f7f6afa353574a2b91da74faefa9831
Author: Olivier Certner <olce@FreeBSD.org>
AuthorDate: 2026-10-13 13:20:43 +0000
Commit: Olivier Certner <olce@FreeBSD.org>
CommitDate: 2026-11-19 15:31:49 +0000
critical_exit_preempt(): Re-check for preemption locked
Once we have the thread lock, we are in a spinlock section and thus will
not be preempted, which ensures stability of 'td_owepreempt'. Check it
again for the improbable case where we were already preempted before
entering that section (after 'td_critnest' reached 0 in critical_exit()
but before clearing 'td_owepreempt', because of an IPI_PREEMPT received
in the meantime or any interrupt whose handler uses critical sections),
in which case there is no need to switch to another thread.
It is however still necessary to temporarily raise 'td_critnest' around
the thread_lock() call, and that is not an optimization contrary to what
the old comment stated. As explained in the new comment, thread_lock()
can lead to entering and exiting a spinlock section multiple times,
which would cause recursion with critical_exit_preempt() in some
circumstances.
Reviewed by: kib, markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D60860
(cherry picked from commit 28c31242758323499eaf8af1f03fc4ae4a960205)
---
sys/kern/kern_switch.c | 28 +++++++++++++++++++---------
1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/sys/kern/kern_switch.c b/sys/kern/kern_switch.c
index 42fee61a99a0..c6f749984217 100644
--- a/sys/kern/kern_switch.c
+++ b/sys/kern/kern_switch.c
@@ -215,12 +215,6 @@ critical_exit_preempt(void)
struct thread *td;
int flags;
- /*
- * If td_critnest is 0, it is possible that we are going to get
- * preempted again before reaching the code below. This happens
- * rarely and is harmless. However, this means td_owepreempt may
- * now be unset.
- */
td = curthread;
if (td->td_critnest != 0)
return;
@@ -228,13 +222,29 @@ critical_exit_preempt(void)
return;
/*
- * Microoptimization: we committed to switch,
- * disable preemption in interrupt handlers
- * while spinning for the thread lock.
+ * Temporarily disable preemption, as thread_lock() can enter/exit
+ * spinlock sections multiple times if the thread lock can't be acquired
+ * right away or changes. A call to spinlock_exit() there causes
+ * a fast-growing stack of recursive calls with critical_exit_preempt()
+ * as long as the condition persists (provided 'td_owepreempt' is still
+ * non-zero), easily leading to stack exhaustion.
*/
td->td_critnest = 1;
thread_lock(td);
td->td_critnest--;
+ /*
+ * We might have been preempted after the test on 'td_critnest' above
+ * and before returning from the thread_lock() call, on reception of an
+ * IPI_PREEMPT or some interrupt whose handler uses critical sections.
+ * If that is the case, 'td_owepreempt' was reset and there is no need
+ * to switch to another thread. We are in a spinlock section here so
+ * 'td_owepreempt' is stable and no incoming preemption will be lost.
+ */
+ if (__predict_false(td->td_owepreempt == 0)) {
+ thread_unlock(td);
+ return;
+ }
+
flags = SW_INVOL | SW_PREEMPT;
if (TD_IS_IDLETHREAD(td))
flags |= SWT_IDLE;