git: 231d75568f16 - main - Move INVLPG to pmap_quick_enter_page() from pmap_quick_remove_page().

The branch main has been updated by kib:

URL: https://cgit.FreeBSD.org/src/commit/?id=231d75568f169fe6adc30373c12512b992c5c672

commit 231d75568f169fe6adc30373c12512b992c5c672
Author:     Konstantin Belousov <kib@FreeBSD.org>
AuthorDate: 2022-12-29 20:48:51 +0000
Commit:     Konstantin Belousov <kib@FreeBSD.org>
CommitDate: 2022-12-31 22:09:46 +0000

    Move INVLPG to pmap_quick_enter_page() from pmap_quick_remove_page().
    
    If processor prefetches neighboring TLB entries to the one being accessed
    (as some have been reported to do), then the spin lock does not prevent
    the situation described in the "AMD64 Architecture Programmer's Manual
    Volume 2: System Programming" rev. 3.23, "7.3.1 Special Coherency
    Considerations".
    
    Reported and reviewed by:       alc
    Sponsored by:   The FreeBSD Foundation
    MFC after:      1 week
    Differential revision:  https://reviews.freebsd.org/D37770
---
 sys/amd64/amd64/pmap.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c
index 07a00963004b..2d2e673e7622 100644
--- a/sys/amd64/amd64/pmap.c
+++ b/sys/amd64/amd64/pmap.c
@@ -10423,6 +10423,13 @@ pmap_quick_enter_page(vm_page_t m)
 		return (PHYS_TO_DMAP(paddr));
 	mtx_lock_spin(&qframe_mtx);
 	KASSERT(*vtopte(qframe) == 0, ("qframe busy"));
+
+	/*
+	 * Since qframe is exclusively mapped by us, and we do not set
+	 * PG_G, we can use INVLPG here.
+	 */
+	invlpg(qframe);
+
 	pte_store(vtopte(qframe), paddr | X86_PG_RW | X86_PG_V | X86_PG_A |
 	    X86_PG_M | pmap_cache_bits(kernel_pmap, m->md.pat_mode, 0));
 	return (qframe);
@@ -10435,14 +10442,6 @@ pmap_quick_remove_page(vm_offset_t addr)
 	if (addr != qframe)
 		return;
 	pte_store(vtopte(qframe), 0);
-
-	/*
-	 * Since qframe is exclusively mapped by
-	 * pmap_quick_enter_page() and that function doesn't set PG_G,
-	 * we can use INVLPG here.
-	 */
-	invlpg(qframe);
-
 	mtx_unlock_spin(&qframe_mtx);
 }