Re: HMP scheduling on FreeBSD

Thursday, January 15, 2026, 10:21:20 PM, you wrote:

> On Wed, Jan 14, 2026 at 04:10:15PM +0000, Minsoo Choo wrote:
>> Greetings,
>> 
>> For the last few days, I've been working on scheduler optimization for heterogeneous cores ("HMP scheduling" from now on) on FreeBSD. After days of reading specs, planning structure, and writing code, I came to a model that can be utilized for implementing HMP scheduling. I opened a first revision on Phabricator (D54674) a few days ago but decided to introduce this model in the mailing list and hear others' opinions.
>> 
>> First of all, HMP-related code (scheduling, provider, cpucap framework, etc) are only built and enabled when "options HMP" is specified in kernel configuration. Of course, this shouldn't be activated by default because it's in experiment status. "options HMP" should be added to kernel configuration to enable this featured and this won't be enabled by default until the majority of developers believe that HMP scheduling is stabilized.
>> 
>> The first component of HMP scheduling is "cpucap". One issue with HMP scheduling is that identifying the capacity and scores of a processor (i.e. providers) is machine-dependent while the scheduler code should be machine-independent, so cpucap acts as an interface between the scheduler and providers. CPU capacity and scores are stored in pcpu structure while the machine's cpucap status (e.g. initialized, has dynamic scores, etc) is stored in global cpucap structure of type "cpucap_t". It also includes functions for scheduler and providers, such as accessors, setters, finding "best" cpu, etc. The review (D54674) adds these facilities under HMP option.
>> 
>> What are capacity and scores? Capacity describes cpu's heterogeneity (more information in sys/contrib/device-tree/Bindings/cpu/cpu-capacity.txt). This is static; it is initialized on boot time (e.g. from device tree) and stored in pc_cap_capacity in pcpu after being normalized to 0-1024. If it is not provided, the default value of 1024 will be used. It is static and core specific; if a performance core has capacity of 1024 and a efficiency core has capacity of 600, this means all performance cores have 1024 and all efficiency cores have 600, and whether they are throttled or not, this information stays the same. For this nature, capacity is hint for loading balancing. By dividing a core's capacity by total capacity, we can assign an equal fraction of tasks to the core's run queue. These cpucap information, including enablement status, has dynamic score, individual capacity and scores, can be queried using sysctl.
>> 
>> On the other hand, scores reflect the real time status of a processor and are normalized to 0-1024. Providers like Intel Thread Director give the current status of each core and feed it to pc_cap_scores in pcpu. Scores are used for thread selection. For example, if a performance core is experiencing throttling, its score could go down to 1000. In that case, the scheduler will prefer core that has the highest score. Scores are completely optional because many processors do not provide this, and when score information is absent, cpucap fall backs to capacity. Since scores are dynamic, they are retrieved and set using atomic operations.
>> 
>> Providers feed capacity or score information to cpucap. Capacity providers such as device trees and ACPI CPPC run on boot time and feed capacity information. If there is no capacity provider, (again) the default value is used. Score providers such as Intel ITD are implemented as device drivers. We had ITD implementation patch (D44453-D44459) back in May 2024 called coredirector, but it was neglected. With some modification, this can be the first cpucap provider. Providers cannot be loaded or unloaded on runtime. An exception is Arm SCMI which doesn't feed information to cpucap, but instead, the scheduler should control and request CPU's performance.
>> 
>> Before integrating scheduler and cpucap, I need to go through sched_ule.c​ from top to bottom. After that, I'll add new functions or drop existing ones from the cpucap framework then work on the integration.
>> 
>> If anyone is interested in this implementation, please reply to this thread and/or review my cpucap patch at D54674.

> Hey Minsoo,

> Thank you very much for working on this! I've merged the patch locally
> on one of my systems running HardenedBSD. I can confirm compilation
> succeeds (and boots).

> I'll be paying attention to this thread for updates. I'm happy to test
> future patch revisions, too, so please reach out even if you want me
> to test a one-line change. I subscribed to patch D54674, so I should
> see updates there.

> One question I have regarding the security posture of heterogeneous
> cores is whether constant-time cryptographic/string/math/etc
> operations take between cores. Would it be possible for an attacker to
> do <something> by virtue of the different types of cores?

> For example, when installing FreeBSD with a geli-backed root
> filesystem, geli determines how quickly cryptographic operations take
> and will adjust the number of rounds accordingly. If that operation
> takes place on an efficiency core, would that result in fewer
> encryption rounds with geli?

> Can some sort of timing oracle exist by virtue of the different cores?
> How are speculative execution vulnerabilities manifest?

> I'm not expecting an answer to any of these questions. I'm just
> writing down some thoughts. But, if anyone can answer those questions,
> all the better. :-)

May I add some other thoughts?

I'd like to see that "nicing"/prioritizing processes will make relative but strict order in a responsiveness/speed of a processes.

I mean that responsiveness/speed in a practice correlated to CPU+IO not CPU time only.

For example, when I run 'dump' server become so slowly that it seem 'hanged' for users.  This is because 'dump' consume little CPU but huge IO so other high(er) priority processes become IO-waiters. No matter how 'dump' niced. In fact higher-priority processes are critically slowly down because of execution of lower-priority process.

May be new scheduler will take into account this problem and provide some hints for a solution.


-- 
Best regards,
Anthony Pankov