[Bug 260805] sysctl shows bogus numa domains

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260805

            Bug ID: 260805
           Summary: sysctl shows bogus numa domains
           Product: Base System
           Version: 12.3-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: pmc@citylink.dinoex.sub.org

Environmental Data:
-------------------
ASUS Z10PA-D8, Cluster-on-Die and NUMA enabled.

CPU: Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (2594.04-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 20 CPUs
FreeBSD/SMP: 1 package(s) x 2 cache groups x 5 core(s) x 2 hardware threads

kern.sched.topology_spec: <groups>
 <group level="1" cache-level="0">
  <cpu count="20" mask="fffff,0,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19</cpu>
  <children>
   <group level="2" cache-level="3">
    <cpu count="10" mask="3ff,0,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9</cpu>
    <children>
     <group level="3" cache-level="2">
      <cpu count="2" mask="3,0,0,0">0, 1</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT
group</flag></flags>
     </group>
     <group level="3" cache-level="2">
      <cpu count="2" mask="c,0,0,0">2, 3</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT
group</flag></flags>
     </group>
     <group level="3" cache-level="2">
      <cpu count="2" mask="30,0,0,0">4, 5</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT
group</flag></flags>
     </group>
[etc.etc.]

This would say that cpu.4 and cpu.5 are SMT brothers on the same processor.
But, when it comes to NUMA domains, sysctl says this:

dev.cpu.19.%domain: 1
dev.cpu.18.%domain: 1
dev.cpu.17.%domain: 1
dev.cpu.16.%domain: 1
dev.cpu.15.%domain: 1
dev.cpu.14.%domain: 0
dev.cpu.13.%domain: 0
dev.cpu.12.%domain: 0
dev.cpu.11.%domain: 0
dev.cpu.10.%domain: 0
dev.cpu.9.%domain: 1
dev.cpu.8.%domain: 1
dev.cpu.7.%domain: 1
dev.cpu.6.%domain: 1
dev.cpu.5.%domain: 1
dev.cpu.4.%domain: 0
dev.cpu.3.%domain: 0
dev.cpu.2.%domain: 0
dev.cpu.1.%domain: 0
dev.cpu.0.%domain: 0

As it is highly unlikely for SMT brothers to share L1/L2 cache but use
different memory banks, something must be wrong here.

I did test with 

cpuset -l 4 -n ft:0-1 sh -c "awk 'END {for(i=0; i < 1000000000; i++) { a[i] =
2.53 } }' < /dev/null"
cpuset -l 5 -n ft:0-1 sh -c "awk 'END {for(i=0; i < 1000000000; i++) { a[i] =
2.53 } }' < /dev/null"

and both do allocate from vm.domain.0.stats.free_count, so it seems the
dev.cpu.X.%domain values are wrong, and there seems no obvious harm resulting
from it (at least until somebody grabs these dev.cpu.X.%domain values to make
their software numa-aware). 

As I have no other numa-capable hardware available, I cannot say if this might
be just an individual flaw in this BIOS, or such.

-- 
You are receiving this mail because:
You are the assignee for the bug.