[Bug 243212] High CPU usage when set affinity on multiple CPU

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243212

            Bug ID: 243212
           Summary: High CPU usage when set affinity on multiple CPU
           Product: Base System
           Version: 12.1-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: pizzamig at freebsd.org

We are facing a minor, but disturbing regression, when we moved from FreeBSD
11.3 to 12.1 on our loadbalancers.

We have 2 identical machines (dual CPU, 4 core per CPU), one running FreeBSD
11.3 and one running 12.1. 
Identical packages and identical configuration.

We have a haproxy instance and, for performance reasons, we pin the threads to
different cpus:
  nbproc 1
  nbthread 6
  cpu-map auto:1/1-6 0-5

The result of this configuration is (on 11.3):
# cpuset -g -p 35799
pid 35799 mask: 0, 1, 2, 3, 4, 5

while on 12.1 is:
# cpuset -g -p 86498
pid 86498 mask: 0, 1, 2, 3, 4, 5
pid 86498 domain policy: first-touch mask: 0, 1

What we have seen is a increased CPU usage (2x) and system load (almost
doubled) on FreeBSD 12.1. We didn't measure a potential impact on the network
latency.
A way to solve/workaround the issue is to decrease the number of threads to 4.

I guess that this regression has something to do with NUMA, maybe threads 4,5
are forced to use memory domain 0, causing the additional CPU usage.

Is this intended? If yes, why on FreeBSD 11.3 we don't see this high CPU usage?
Thanks in advance for the help

I report here some hardware information.

hw.model: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz
hw.ncpu: 8
kern.sched.topology_spec: <groups>
 <group level="1" cache-level="0">
  <cpu count="8" mask="ff,0,0,0">0, 1, 2, 3, 4, 5, 6, 7</cpu>
  <children>
   <group level="2" cache-level="3">
    <cpu count="4" mask="f,0,0,0">0, 1, 2, 3</cpu>
    <children>
     <group level="3" cache-level="2">
      <cpu count="1" mask="1,0,0,0">0</cpu>
     </group>
     <group level="3" cache-level="2">
      <cpu count="1" mask="2,0,0,0">1</cpu>
     </group>
     <group level="3" cache-level="2">
      <cpu count="1" mask="4,0,0,0">2</cpu>
     </group>
     <group level="3" cache-level="2">
      <cpu count="1" mask="8,0,0,0">3</cpu>
     </group>
    </children>
   </group>
   <group level="2" cache-level="3">
    <cpu count="4" mask="f0,0,0,0">4, 5, 6, 7</cpu>
    <children>
     <group level="3" cache-level="2">
      <cpu count="1" mask="10,0,0,0">4</cpu>
     </group>
     <group level="3" cache-level="2">
      <cpu count="1" mask="20,0,0,0">5</cpu>
     </group>
     <group level="3" cache-level="2">
      <cpu count="1" mask="40,0,0,0">6</cpu>
     </group>
     <group level="3" cache-level="2">
      <cpu count="1" mask="80,0,0,0">7</cpu>
     </group>
    </children>
   </group>
  </children>
 </group>
</groups>

-- 
You are receiving this mail because:
You are the assignee for the bug.