svn commit: r347957 - head/sys/amd64/amd64

[Konstantin Belousov]

Author: kib
Date: Sat May 18 16:19:31 2019
New Revision: 347957
URL: https://svnweb.freebsd.org/changeset/base/347957

Log:
  Make lock-less delayed invalidation operational very early.
  
  Apparently, there is more code trying to call pmap_remove() early,
  mostly to free preloaded memory.  Instead of moving all deallocations
  to the point where a scheduler is initialized, add missed setup of
  thread0 di init at hammer_time().
  
  The code in pmap_delayed_invl_start_u() is modified to not ever take
  the thread lock if the thread priority is less or equal to PVM.  Since
  thread0 starts at priority 0, and then is reset to PVM at
  proc0_init(), this eliminates taking the thread lock during early
  boot.
  
  While there, fix off by one in comparision of the base priority.
  
  Reported and tested by:	bcran (previous version)
  Reviewed by:	markj
  Sponsored by:	The FreeBSD Foundation
  MFC after:	29 days

Modified:
  head/sys/amd64/amd64/machdep.c
  head/sys/amd64/amd64/pmap.c

Modified: head/sys/amd64/amd64/machdep.c
==============================================================================
--- head/sys/amd64/amd64/machdep.c	Sat May 18 14:55:59 2019	(r347956)
+++ head/sys/amd64/amd64/machdep.c	Sat May 18 16:19:31 2019	(r347957)
@@ -1617,6 +1617,13 @@ hammer_time(u_int64_t modulep, u_int64_t physfree)
 	physfree += kstack0_sz;
 
 	/*
+	 * Initialize enough of thread0 for delayed invalidation to
+	 * work very early.  Rely on thread0.td_base_pri
+	 * zero-initialization, it is reset to PVM at proc0_init().
+	 */
+	pmap_thread_init_invl_gen(&thread0);
+
+	/*
 	 * make gdt memory segments
 	 */
 	for (x = 0; x < NGDT; x++) {

Modified: head/sys/amd64/amd64/pmap.c
==============================================================================
--- head/sys/amd64/amd64/pmap.c	Sat May 18 14:55:59 2019	(r347956)
+++ head/sys/amd64/amd64/pmap.c	Sat May 18 16:19:31 2019	(r347957)
@@ -700,16 +700,17 @@ pmap_delayed_invl_start_u(void)
 	invl_gen = &td->td_md.md_invl_gen;
 	PMAP_ASSERT_NOT_IN_DI();
 	lock_delay_arg_init(&lda, &di_delay);
-	thread_lock(td);
+	invl_gen->saved_pri = 0;
 	pri = td->td_base_pri;
-	if (pri < PVM) {
-		invl_gen->saved_pri = 0;
-	} else {
-		invl_gen->saved_pri = pri;
-		sched_prio(td, PVM);
+	if (pri > PVM) {
+		thread_lock(td);
+		pri = td->td_base_pri;
+		if (pri > PVM) {
+			invl_gen->saved_pri = pri;
+			sched_prio(td, PVM);
+		}
+		thread_unlock(td);
 	}
-	thread_unlock(td);
-
 again:
 	PV_STAT(i = 0);
 	for (p = &pmap_invl_gen_head;; p = prev.next) {