svn commit: r357014 - head/sys/kern

[Jeff Roberson]

Author: jeff
Date: Thu Jan 23 03:36:50 2020
New Revision: 357014
URL: https://svnweb.freebsd.org/changeset/base/357014

Log:
  Block the thread lock in sched_throw() and use cpu_switch() to unblock
  it.  The introduction of lockless switch in r355784 created a race to
  re-use the exiting thread that was only possible to hit on a hypervisor.
  
  Reported/Tested by:	rlibby
  Discussed with:	rlibby, jhb

Modified:
  head/sys/kern/sched_ule.c

Modified: head/sys/kern/sched_ule.c
==============================================================================
--- head/sys/kern/sched_ule.c	Thu Jan 23 01:49:22 2020	(r357013)
+++ head/sys/kern/sched_ule.c	Thu Jan 23 03:36:50 2020	(r357014)
@@ -2894,7 +2894,7 @@ sched_throw(struct thread *td)
 	struct thread *newtd;
 	struct tdq *tdq;
 
-	if (td == NULL) {
+	if (__predict_false(td == NULL)) {
 #ifdef SMP
 		PCPU_SET(sched, DPCPU_PTR(tdq));
 #endif
@@ -2912,13 +2912,18 @@ sched_throw(struct thread *td)
 		tdq_load_rem(tdq, td);
 		td->td_lastcpu = td->td_oncpu;
 		td->td_oncpu = NOCPU;
+		thread_lock_block(td);
 	}
 	newtd = choosethread();
 	spinlock_enter();
 	TDQ_UNLOCK(tdq);
 	KASSERT(curthread->td_md.md_spinlock_count == 1,
 	    ("invalid count %d", curthread->td_md.md_spinlock_count));
-	cpu_throw(td, newtd);		/* doesn't return */
+	/* doesn't return */
+	if (__predict_false(td == NULL))
+		cpu_throw(td, newtd);		/* doesn't return */
+	else
+		cpu_switch(td, newtd, TDQ_LOCKPTR(tdq));
 }
 
 /*