Optimize execution of processes by CPU core

[Eugene Grosbein]

21.02.2019 0:35, Farhan Khan via freebsd-hackers wrote:
> Hi all,
> 
> I am trying to optimize the execution of a CPU-intensive workload where I am running multiple instances of a program.
> The moment the program ends (expected behavior), the calling shell script verifies the results and if its good it reruns the program.
> The machines I am running this on have 8 cores, but ps reports that some of the processes frequently run on the same CPU,
> so I suspect I am not getting optimized performance.

System scheduler switches useland processed from one CPU core to another often enough
so each CPU has even load and you cannot see that with naked. But you can easily verify
if load is even or not checking sysctl kern.cp_times that shows five monotonically increasing
counters per each CPU core. For example, in case of dual-core system:

$ sysctl kern.cp_times
kern.cp_times: 14789486 132229 14016113 327160 949428773 14374865 139326 12056998 2941012 949179638
$ sysctl kern.clockrate
kern.clockrate: { hz = 1000, tick = 1000, profhz = 8126, stathz = 127 }

There are "stathz" ticks per second and for each core exactly one of five counters is incremented by one:
user, nice, system, interrupt, idle. That is, each 5th counter is incremented
if corresponding CPU core was idle during "the tick".

You can save output of sysctl kern.cp_times, run your test, stop it and save output again.
Then compare difference of each 5th counter and they should be approximately equal.

You can even draw graphs if you periodically get samples of the sysctl, compute diffs with previous samples,
divide diffs by period length in seconds and then divide again by "stathz" value.
Multiply by 100 to get idle time of single CPU code in percents for the period.
Repeat for each core.

I use net-mgmt/mrtg to draw such per-CPU graphs for my servers, it works just fine.