Re: Current ULE state and plans (was Re: HMP scheduling on FreeBSD)
- Reply: Mark Millard : "Re: Current ULE state and plans (was Re: HMP scheduling on FreeBSD)"
- Reply: Olivier Certner : "Re: Current ULE state and plans (was Re: HMP scheduling on FreeBSD)"
- In reply to: Olivier Certner : "Re: Current ULE state and plans (was Re: HMP scheduling on FreeBSD)"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Fri, 30 Jan 2026 21:21:00 UTC
On 1/22/26 12:49, Olivier Certner wrote:
> Hi,
>
>> Since we seem to need a "count" on useful use of old code, add me. Also
>> IIRC, bde had several issues with ULE.
>
> That's not much. Again, we need scenarios, characterization, numbers, etc. Not going to repeat myself ever and ever.
All,
I rebuilt world after the series of commits from kib@
to allow 4BSD and ULE to exist in the same kernel.
TL;DR version: In an over-subscribed HPC situation 4BSD
performs better than ULE, but ULE is better than its
historical performance from a decade ago.
% uname -a
FreeBSD hotrats 16.0-CURRENT FreeBSD 16.0-CURRENT #4
main-n283502-fdcd67be8274: Thu Jan 29 17:55:38 PST 2026
kargl@hotrats:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
% lscpu
Architecture: amd64
Byte Order: Little Endian
Total CPU(s): 16
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Vendor: AuthenticAMD
CPU family: 25
Model: 97
Model name: AMD Ryzen 7 7700X 8-Core Processor
Stepping: 2
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 32M
I have rerun a number of numerically intensive loading tests,
where I use MPI. Specifically, I have installed
% pkg info | grep openmpi
openmpi-5.0.9_1 High Performance Message Passing Library
% pkg info | grep gcc
gcc15-15.2.0_1 GNU Compiler Collection 15
The task that I will be using is a classic boss-worker (almost
perfectly) parallel algorithm. There is a bit of initialization
overhead, a small amount of communication between the boss
and a worker image, and overhead for teardown. Otherwise, the
workers dominate the overall time.
A small caveat about openmpi. It has been updated to try to
prevent an over-subscribed HPC situation where it checks
the node for the number of physical cores. This of course
can only prevent a single user from loading a node. If multiple
people are using a HPC system, one can still cause issues.
% mpirun -n 17 --hostfile mf ../../build/bin/reverbmp reverbmp.in
--------------------------------------------------------------------------
A request was made to bind that would require binding
processes to more cpus than are available in your allocation:
Application: ../../build/bin/reverbmp
#processes: 17
Mapping policy: BYCORE
Binding policy: CORE
You can override this protection by adding the "overload-allowed"
option to your binding directive.
--------------------------------------------------------------------------
One can prevent the above error with the '--bind-to none' option.
Note, the machinefile contains the single line 'localhost slots=17'.
On a freshly booted and lightly loaded system
% sysctl -a | grep ULE
kern.sched.available: 4BSD,ULE
kern.sched.name: ULE
% mpirun -n 8 --hostfile mf ../../build/bin/reverbmp reverbmp.in
...
Wall time: 15.191 min
CPU time: 15.191 min
last pid: 3524; load averages: 8.00, 7.66, 5 up 0+00:23:37
11:07:32
58 processes: 9 running, 49 sleeping
CPU: 50.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 50.0% idle
Mem: 1731M Active, 9580K Inact, 1079M Wired, 562M Buf, 59G Free
Swap: 16G Total, 16G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU
COMMAND
3473 kargl 2 141 0 656M 298M CPU0 0 15:11 99.99%
reverbmp
3480 kargl 2 141 0 400M 205M CPU14 14 15:11 99.99%
reverbmp
3476 kargl 2 141 0 400M 205M CPU6 6 15:11 99.99%
reverbmp
3477 kargl 2 141 0 400M 205M CPU8 8 15:11 99.99%
reverbmp
3474 kargl 2 141 0 400M 205M CPU2 2 15:11 99.99%
reverbmp
3479 kargl 2 141 0 400M 205M CPU12 12 15:11 99.99%
reverbmp
3475 kargl 2 141 0 400M 205M CPU4 4 15:11 99.99%
reverbmp
3478 kargl 2 141 0 400M 205M CPU10 10 15:11 99.99%
reverbmp
% mpirun -n 16 --bind-to none --hostfile mf ../../build/bin/reverbmp
reverbmp.in
...
Wall time: 13.905 min
CPU time: 13.904 min
top(1) is similar to the above '-n 8' case except 16 images are running.
The following is over-subscribed.
% mpirun -n 17 --bind-to none --hostfile mf ../../build/bin/reverbmp
reverbmp.in
...
Wall time: 16.456 min <--- Whoops.
CPU time: 15.358 min
last pid: 3611; load averages: 17.01, 16.78, 14 up 0+00:54:07
11:38:02
67 processes: 18 running, 49 sleeping
CPU: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
Mem: 3384M Active, 35M Inact, 1149M Wired, 614M Buf, 57G Free
Swap: 16G Total, 16G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU
COMMAND
3597 kargl 2 141 0 544M 208M CPU11 11 12:26 100.68%
reverbmp
3590 kargl 2 137 0 544M 208M RUN 0 12:22 95.39%
reverbmp
3596 kargl 2 135 0 544M 208M CPU13 13 12:21 100.68%
reverbmp
3598 kargl 2 140 0 544M 208M CPU14 14 12:21 100.68%
reverbmp
3587 kargl 2 141 0 552M 208M CPU12 12 12:21 100.67%
reverbmp
3588 kargl 2 136 0 544M 208M CPU0 0 12:21 51.95%
reverbmp
3592 kargl 2 134 0 544M 208M CPU15 15 12:18 100.68%
reverbmp
3595 kargl 2 139 0 544M 208M CPU4 4 12:17 100.68%
reverbmp
3599 kargl 2 141 0 544M 208M CPU10 10 12:16 100.66%
reverbmp
3584 kargl 2 140 0 544M 208M CPU3 3 12:16 100.68%
reverbmp
3593 kargl 2 141 0 544M 208M RUN 1 12:10 99.96%
reverbmp
3589 kargl 2 133 0 544M 208M CPU8 8 12:10 53.45%
reverbmp
3591 kargl 2 140 0 544M 208M CPU5 5 12:10 100.68%
reverbmp
3594 kargl 2 140 0 544M 208M CPU6 6 12:09 100.68%
reverbmp
3586 kargl 2 131 0 544M 208M CPU2 2 12:09 100.68%
reverbmp
3583 kargl 2 140 0 800M 280M CPU9 9 12:09 100.68%
reverbmp
3585 kargl 2 141 0 552M 208M CPU7 7 12:04 100.68%
reverbmp
Note, pid 3583 is the boss and I've sorted top(1) by time.
Two observations: (1) the time accumulation for the images is
skewed, which suggests load balancing is still an issue. (2) The
% CPU suggests that processes can be penalized while switching
among the cpus.
Now, let's reboot with 4BSD.
% sysctl -a | grep 4BSD
kern.sched.available: 4BSD,ULE
kern.sched.name: 4BSD
% mpirun -n 8 --hostfile mf ../../build/bin/reverbmp reverbmp.in
...
Wall time: 15.320 min
CPU time: 15.320 min
% mpirun -n 16 --bind-to none --hostfile mf ../../build/bin/reverbmp
reverbmp.in
...
Wall time: 13.955 min
CPU time: 13.941 min
The top(1) output is similar to the above ULE cases with
'-n 8' and '-n 16'. That is, the workers all have 99.xx%
CPU.
% mpirun -n 17 --bind-to none --hostfile mf ../../build/bin/reverbmp
reverbmp.in
....
Wall time: 13.893 min
CPU time: 13.072 min
last pid: 2972; load averages: 17.09, 16.82, 14 up 0+00:46:02
12:31:04
66 processes: 18 running, 48 sleeping
CPU: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
Mem: 3382M Active, 87M Inact, 841M Wired, 298M Buf, 58G Free
Swap: 16G Total, 16G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU
COMMAND
2951 kargl 2 52 0 552M 207M CPU8 8 11:24 94.33%
reverbmp
2944 kargl 2 52 0 544M 207M CPU7 7 11:24 94.33%
reverbmp
2954 kargl 2 52 0 552M 207M CPU3 3 11:24 93.53%
reverbmp
2946 kargl 2 52 0 544M 207M CPU11 11 11:24 92.73%
reverbmp
2958 kargl 2 52 0 544M 207M CPU10 10 11:24 91.97%
reverbmp
2942 kargl 2 52 0 800M 286M RUN 6 11:24 97.48%
reverbmp
2957 kargl 2 52 0 544M 207M CPU15 15 11:24 92.75%
reverbmp
2948 kargl 2 52 0 544M 207M CPU5 5 11:24 99.04%
reverbmp
2949 kargl 2 52 0 544M 207M CPU6 6 11:24 92.76%
reverbmp
2945 kargl 2 52 0 544M 207M CPU12 12 11:24 92.76%
reverbmp
2943 kargl 2 52 0 544M 207M CPU4 4 11:24 96.69%
reverbmp
2947 kargl 2 52 0 544M 207M CPU1 1 11:24 95.90%
reverbmp
2955 kargl 2 52 0 544M 207M CPU13 13 11:24 97.47%
reverbmp
2956 kargl 2 52 0 544M 207M CPU9 9 11:24 92.04%
reverbmp
2950 kargl 2 52 0 544M 207M RUN 2 11:24 95.90%
reverbmp
2952 kargl 2 52 0 544M 207M CPU0 0 11:24 96.67%
reverbmp
2953 kargl 2 52 0 544M 207M CPU14 14 11:24 92.73%
reverbmp
The image with lowest pid is the boss, i.e., pid 2942. The
accumulated time and % CPU numbers suggest a round-robin like
scheduling. More importantly, one see
ULE:
Wall time: 16.456 min
CPU time: 15.358 min
4BSD:
Wall time: 13.893 min
CPU time: 13.072 min
One of these is better from a HPC viewpoint.
--
steve