Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

On Tue, Jan 03, 2023 at 08:20:16AM +0000, Chen, Alvin W wrote:
> > 
> > [EXTERNAL EMAIL]
> > 
> > On 2022-11-14 09:09 +0200, Konstantin Belousov wrote:
> > >
> > > You might use this patch meantime
> > > https://urldefense.com/v3/__https://kib.kiev.ua/git/gitweb.cgi?p=devia
> > >
> > nt3.git;a=commit;h=5d72240a8777b26d5e0a7d2d26bb919d05f60002__;!!Lp
> > KI!j
> > >
> > pyHChyB8NZAQq5isiNFepD61cX0HczrFCeOriYbSwyfMPGW7k8I_BYxPWXv1FG
> > yfl1Y-Ip
> > > dgcNkyg$ [kib[.]kiev[.]ua]
> > 
> > Thank you for this patch.  On my 13.1 machine it fixed all the panics with a i9-
> > 12900KF.  It's nice to be able to use all the cores since I got this machine in
> > May.  I had them disabled in BIOS.
> > 
> > Also as a side note this fixed the audio issues I was having with the e cores
> > both enabled and disabled.
> > 
> 
> This patch can't work well with some older CPUs, and I find one N6005 (Intel JSP).
> This JSP cpu is not a hybrid cpu architecture, but the high cpu id is 0x1b with 0x20000000 which meet the small core condition in the patch, but kernel got crashed on this cpu.
> We need a better condition to pick up the E cores to make sure the kernel can be compatible with all older Non-Hybrid CPUs.

How does the kernel panic?  Could you show the verbose dmesg lines with
CPUID information?

If this is JasperLake, I suspect that the patch below should be enough.

Still, the workaround is not confirmed/explained by you (Intel).  I have
a high hope that we would eventually be able to use INVLPG with PCID on
(newer) small cores, might be even on older small cores after a microcode
update.

commit c8812792202d73804cccd624f6478c4069b22438
Author: Konstantin Belousov <kib@FreeBSD.org>
Date:   Tue Jan 3 12:13:07 2023 +0200

    amd64: be more precise when enabling the AlderLake workaround
    
    Reported by:    "Chen, Alvin W" <Weike.Chen@Dell.com>
    Sponsored by:   The FreeBSD Foundation
    MFC after:      1 week

diff --git a/sys/amd64/amd64/initcpu.c b/sys/amd64/amd64/initcpu.c
index 08385d3095d0..cddf8502437e 100644
--- a/sys/amd64/amd64/initcpu.c
+++ b/sys/amd64/amd64/initcpu.c
@@ -247,6 +247,26 @@ cpu_auxmsr(void)
 	return (PCPU_GET(cpuid));
 }
 
+void
+cpu_init_small_core(void)
+{
+	u_int r[4];
+
+	if (cpu_high < 0x1a)
+		return;
+
+	cpuid_count(0x1a, 0, r);
+	if ((r[0] & CPUID_HYBRID_CORE_MASK) != CPUID_HYBRID_SMALL_CORE)
+		return;
+
+	PCPU_SET(small_core, 1);
+	if (pmap_pcid_enabled && invpcid_works &&
+	    pmap_pcid_invlpg_workaround_uena) {
+		PCPU_SET(pcid_invlpg_workaround, 1);
+		pmap_pcid_invlpg_workaround = 1;
+	}
+}
+
 /*
  * Initialize CPU control registers
  */
@@ -255,7 +275,6 @@ initializecpu(void)
 {
 	uint64_t msr;
 	uint32_t cr4;
-	u_int r[4];
 
 	cr4 = rcr4();
 	if ((cpu_feature & CPUID_XMM) && (cpu_feature & CPUID_FXSR)) {
@@ -319,18 +338,8 @@ initializecpu(void)
 	    (cpu_stdext_feature2 & CPUID_STDEXT2_RDPID) != 0)
 		wrmsr(MSR_TSC_AUX, cpu_auxmsr());
 
-	if (cpu_high >= 0x1a) {
-		cpuid_count(0x1a, 0, r);
-		if ((r[0] & CPUID_HYBRID_CORE_MASK) ==
-		    CPUID_HYBRID_SMALL_CORE) {
-			PCPU_SET(small_core, 1);
-			if (pmap_pcid_enabled &&
-			    pmap_pcid_invlpg_workaround_uena) {
-				PCPU_SET(pcid_invlpg_workaround, 1);
-				pmap_pcid_invlpg_workaround = 1;
-			}
-		}
-	}
+	if (!IS_BSP())
+		cpu_init_small_core();
 }
 
 void
diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c
index 05342b31d2aa..c601ce868978 100644
--- a/sys/amd64/amd64/machdep.c
+++ b/sys/amd64/amd64/machdep.c
@@ -1341,6 +1341,12 @@ hammer_time(u_int64_t modulep, u_int64_t physfree)
 		pmap_pcid_enabled = 0;
 	}
 
+	/*
+	 * Now we can do small core initialization, after the PCID
+	 * CPU features and user knobs are evaluated.
+	 */
+	cpu_init_small_core();
+
 	link_elf_ireloc(kmdp);
 
 	/*
diff --git a/sys/amd64/include/md_var.h b/sys/amd64/include/md_var.h
index f014c66c0d06..f5cbdb6bbd9d 100644
--- a/sys/amd64/include/md_var.h
+++ b/sys/amd64/include/md_var.h
@@ -72,6 +72,7 @@ void	amd64_bsp_ist_init(struct pcpu *pc);
 void	amd64_syscall(struct thread *td, int traced);
 void	amd64_syscall_ret_flush_l1d(int error);
 void	amd64_syscall_ret_flush_l1d_recalc(void);
+void	cpu_init_small_core(void);
 void	doreti_iret(void) __asm(__STRING(doreti_iret));
 void	doreti_iret_fault(void) __asm(__STRING(doreti_iret_fault));
 void	flush_l1d_sw_abi(void);