8.1-PRERELEASE: CPU packages not detected correctly

Fri Jul 16 15:48:04 UTC 2010

On Thursday 15 July 2010 09:34 pm, David Xu wrote:
> Jung-uk Kim wrote:
> > On Wednesday 14 July 2010 05:40 pm, Jung-uk Kim wrote:
> >> On Wednesday 14 July 2010 01:31 pm, Andriy Gapon wrote:
> >>> on 14/07/2010 17:14 Oliver Fromme said the following:
> >>>> In a machine installed yesterday, 8.1-PRERELEASE doesn't
> >>>> seem to detect the number of CPU packages vs. cores per
> >>>>
> >>>> package correctly:
> >>>>  | FreeBSD 8.1-PRERELEASE-20100713 #0: Tue Jul 13 19:51:18 UTC
> >>>>  | 2010 [...]
> >>>>  | CPU: Intel(R) Xeon(R) CPU           L5408  @ 2.13GHz
> >>>>  | (2133.42-MHz K8-class CPU) Origin = "GenuineIntel"  Id =
> >>>>  | 0x1067a  Family = 6  Model = 17  Stepping = 10
> >>>>  | Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC
> >>>>  |, SE
> >>>>  | P,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE
> >>>>  |, SSE 2,SS,HTT,TM,PBE>
> >>>>  | Features2=0x40ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE
> >>>>  |3 ,C X16,xTPR,PDCM,DCA,SSE4.1,XSAVE> AMD
> >>>>  | Features=0x20000800<SYSCALL,LM>
> >>>>  |   AMD Features2=0x1<LAHF>
> >>>>  |   TSC: P-state invariant
> >>>>  | real memory  = 34359738368 (32768 MB)
> >>>>  | avail memory = 33151377408 (31615 MB)
> >>>>  | ACPI APIC Table: <IBM    SERBLADE>
> >>>>  | FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
> >>>>  | FreeBSD/SMP: 1 package(s) x 8 core(s)
> >>>>  |  cpu0 (BSP): APIC ID:  0
> >>>>  |  cpu1 (AP): APIC ID:  1
> >>>>  |  cpu2 (AP): APIC ID:  2
> >>>>  |  cpu3 (AP): APIC ID:  3
> >>>>  |  cpu4 (AP): APIC ID:  4
> >>>>  |  cpu5 (AP): APIC ID:  5
> >>>>  |  cpu6 (AP): APIC ID:  6
> >>>>  |  cpu7 (AP): APIC ID:  7
> >>>>  | ioapic1 <Version 2.0> irqs 24-47 on motherboard
> >>>>  | ioapic0 <Version 2.0> irqs 0-23 on motherboard
> >>>>
> >>>> I'm pretty sure that this is a 2 x 4 machine (2 CPU packages
> >>>> with 4 cores per package), not 1 x 8.  That's what the BIOS
> >>>> displays during POST.
> >>>>
> >>>> I'm not sure if this is just a "cosmetic" issue, or if this
> >>>> is a critical thing ...  I could imagine that performance
> >>>> might be sub-optimal if the CPU topology isn't detected
> >>>> correctly, but I'm not sure if FreeBSD can take advantage
> >>>> of the topology.
> >>>
> >>> Could you please try to do the following?
> >>> 1. Fetch topo-12212009.tar from the top of this page:
> >>> http://software.intel.com/en-us/articles/intel-64-architecture-
> >>>pr oc essor-topology-enumeration/ 2. Untar it and apply this
> >>> patch to the code:
> >>> http://people.freebsd.org/~avg/cpu-topology.diff
> >>> 3. Compile it by running sh mk_64.sh (supposing you have amd64
> >>> system installed) 4. Run cpu_topology64.out and report back its
> >>> output.
> >>
> >> It's funny that I actually wrote a convenience script (and
> >> cleaned up today):
> >>
> >> http://people.freebsd.org/~jkim/cpu_topology-12212009.sh
> >>
> >> BTW, current topology detection code is not optimal for some
> >> Intel processors if my memory serves.
> >
> > Surprisingly, I found a patch I wrote last year from /tmp of my
> > desktop and it is still applied cleanly. 8-)
> >
> > Just in case, it is available from here:
> >
> > http://people.freebsd.org/~jkim/mp_machdep.diff
> >
> > It is applicable to both sys/amd64/amd64/mp_machdep.c and
> > sys/i386/i386/mp_machdep.c.  I have to warn you, though, it
> > hasn't been used for more than a year, so it may not work at all.
> > ;-)
> >
> > Jung-uk Kim
>
> Do you have patch for i386 branch ? I want to test.

The patch should apply fine on both sys/amd64/amd64/mp_machdep.c and 
sys/i386/i386/mp_machdep.c.

http://people.freebsd.org/~jkim/mp_machdep2.diff

> On my Pentium-D machine:
>
> $ sysctl kern.sched.topology_spec
> kern.sched.topology_spec: <groups>
>   <group level="1" cache-level="0">
>    <cpu count="2" mask="0x3">0, 1</cpu>
>    <flags></flags>
>    <children>
>     <group level="3" cache-level="2">
>      <cpu count="2" mask="0x3">0, 1</cpu>
>      <flags></flags>
>     </group>
>    </children>
>   </group>
> </groups>
>
> it seems the kernel thinks that the Pentuim-D is sharing L2 cache,
> but in fact, the design of the Pentium Ds was simply two P4 cores
> sitting side by side. They do not share anything and they basically
> work independently.

cpu_topo() in mp_machdep.c is not that smart to distinguish whether 
two cores in one package is sharing L2 cache or not:

	/*
	 * Only HTT no multi-core.
	 */
	if (cpu_logical > 1 && cpu_cores == 1)
		return (smp_topo_1level(CG_SHARE_L1, cpu_logical, cg_flags));
	/*
	 * Only multi-core no HTT.
	 */
	if (cpu_cores > 1 && cpu_logical == 1)
		return (smp_topo_1level(CG_SHARE_L2, cpu_cores, cg_flags));
	/*
	 * Both HTT and multi-core.
	 */
	return (smp_topo_2level(CG_SHARE_L2, cpu_cores,
	    CG_SHARE_L1, cpu_logical, cg_flags));

In other words, if one package contains multiple cores, it is 
*assumed* sharing L2 cache.  The same is true for HTT/SMT case (i.e., 
assumed sharing L1 cache).  It does not care about L3 cache at all.

Yes, we need some improvement in this area.

Jung-uk Kim