Re: Current ULE state and plans (was Re: HMP scheduling on FreeBSD)
- In reply to: Steve Kargl : "Re: Current ULE state and plans (was Re: HMP scheduling on FreeBSD)"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sat, 31 Jan 2026 01:43:21 UTC
On 1/30/26 13:21, Steve Kargl wrote:
> On 1/22/26 12:49, Olivier Certner wrote:
>> Hi,
>>
>>> Since we seem to need a "count" on useful use of old code, add me. Also
>>> IIRC, bde had several issues with ULE.
>>
>> That's not much. Again, we need scenarios, characterization, numbers,
>> etc. Not going to repeat myself ever and ever.
>
> All,
>
> I rebuilt world after the series of commits from kib@
> to allow 4BSD and ULE to exist in the same kernel.
>
> TL;DR version: In an over-subscribed HPC situation 4BSD
> performs better than ULE, but ULE is better than its
> historical performance from a decade ago.
>
> % uname -a
> FreeBSD hotrats 16.0-CURRENT FreeBSD 16.0-CURRENT #4
> main-n283502-fdcd67be8274: Thu Jan 29 17:55:38 PST 2026
> kargl@hotrats:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
>
> % lscpu
> Architecture: amd64
> Byte Order: Little Endian
> Total CPU(s): 16
> Thread(s) per core: 2
> Core(s) per socket: 8
> Socket(s): 1
> Vendor: AuthenticAMD
> CPU family: 25
> Model: 97
> Model name: AMD Ryzen 7 7700X 8-Core Processor
> Stepping: 2
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 1024K
> L3 cache: 32M
>
>
> I have rerun a number of numerically intensive loading tests,
> where I use MPI. Specifically, I have installed
>
> % pkg info | grep openmpi
> openmpi-5.0.9_1 High Performance Message Passing Library
> % pkg info | grep gcc
> gcc15-15.2.0_1 GNU Compiler Collection 15
>
> The task that I will be using is a classic boss-worker (almost
> perfectly) parallel algorithm. There is a bit of initialization
> overhead, a small amount of communication between the boss
> and a worker image, and overhead for teardown. Otherwise, the
> workers dominate the overall time.
>
> A small caveat about openmpi. It has been updated to try to
> prevent an over-subscribed HPC situation where it checks
> the node for the number of physical cores. This of course
> can only prevent a single user from loading a node. If multiple
> people are using a HPC system, one can still cause issues.
>
> % mpirun -n 17 --hostfile mf ../../build/bin/reverbmp reverbmp.in
> --------------------------------------------------------------------------
> A request was made to bind that would require binding
> processes to more cpus than are available in your allocation:
>
> Application: ../../build/bin/reverbmp
> #processes: 17
> Mapping policy: BYCORE
> Binding policy: CORE
>
> You can override this protection by adding the "overload-allowed"
> option to your binding directive.
> --------------------------------------------------------------------------
>
> One can prevent the above error with the '--bind-to none' option.
> Note, the machinefile contains the single line 'localhost slots=17'.
>
> On a freshly booted and lightly loaded system
>
> % sysctl -a | grep ULE
> kern.sched.available: 4BSD,ULE
> kern.sched.name: ULE
>
> % mpirun -n 8 --hostfile mf ../../build/bin/reverbmp reverbmp.in
> ...
> Wall time: 15.191 min
> CPU time: 15.191 min
>
> last pid: 3524; load averages: 8.00, 7.66, 5 up 0+00:23:37
> 11:07:32
> 58 processes: 9 running, 49 sleeping
> CPU: 50.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 50.0% idle
> Mem: 1731M Active, 9580K Inact, 1079M Wired, 562M Buf, 59G Free
> Swap: 16G Total, 16G Free
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU
> COMMAND
> 3473 kargl 2 141 0 656M 298M CPU0 0 15:11 99.99%
> reverbmp
> 3480 kargl 2 141 0 400M 205M CPU14 14 15:11 99.99%
> reverbmp
> 3476 kargl 2 141 0 400M 205M CPU6 6 15:11 99.99%
> reverbmp
> 3477 kargl 2 141 0 400M 205M CPU8 8 15:11 99.99%
> reverbmp
> 3474 kargl 2 141 0 400M 205M CPU2 2 15:11 99.99%
> reverbmp
> 3479 kargl 2 141 0 400M 205M CPU12 12 15:11 99.99%
> reverbmp
> 3475 kargl 2 141 0 400M 205M CPU4 4 15:11 99.99%
> reverbmp
> 3478 kargl 2 141 0 400M 205M CPU10 10 15:11 99.99%
> reverbmp
>
> % mpirun -n 16 --bind-to none --hostfile mf ../../build/bin/reverbmp
> reverbmp.in
> ...
> Wall time: 13.905 min
> CPU time: 13.904 min
>
> top(1) is similar to the above '-n 8' case except 16 images are running.
> The following is over-subscribed.
>
> % mpirun -n 17 --bind-to none --hostfile mf ../../build/bin/reverbmp
> reverbmp.in
> ...
> Wall time: 16.456 min <--- Whoops.
> CPU time: 15.358 min
>
> last pid: 3611; load averages: 17.01, 16.78, 14 up 0+00:54:07
> 11:38:02
> 67 processes: 18 running, 49 sleeping
> CPU: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
> Mem: 3384M Active, 35M Inact, 1149M Wired, 614M Buf, 57G Free
> Swap: 16G Total, 16G Free
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU
> COMMAND
> 3597 kargl 2 141 0 544M 208M CPU11 11 12:26 100.68%
> reverbmp
> 3590 kargl 2 137 0 544M 208M RUN 0 12:22 95.39%
> reverbmp
> 3596 kargl 2 135 0 544M 208M CPU13 13 12:21 100.68%
> reverbmp
> 3598 kargl 2 140 0 544M 208M CPU14 14 12:21 100.68%
> reverbmp
> 3587 kargl 2 141 0 552M 208M CPU12 12 12:21 100.67%
> reverbmp
> 3588 kargl 2 136 0 544M 208M CPU0 0 12:21 51.95%
> reverbmp
> 3592 kargl 2 134 0 544M 208M CPU15 15 12:18 100.68%
> reverbmp
> 3595 kargl 2 139 0 544M 208M CPU4 4 12:17 100.68%
> reverbmp
> 3599 kargl 2 141 0 544M 208M CPU10 10 12:16 100.66%
> reverbmp
> 3584 kargl 2 140 0 544M 208M CPU3 3 12:16 100.68%
> reverbmp
> 3593 kargl 2 141 0 544M 208M RUN 1 12:10 99.96%
> reverbmp
> 3589 kargl 2 133 0 544M 208M CPU8 8 12:10 53.45%
> reverbmp
> 3591 kargl 2 140 0 544M 208M CPU5 5 12:10 100.68%
> reverbmp
> 3594 kargl 2 140 0 544M 208M CPU6 6 12:09 100.68%
> reverbmp
> 3586 kargl 2 131 0 544M 208M CPU2 2 12:09 100.68%
> reverbmp
> 3583 kargl 2 140 0 800M 280M CPU9 9 12:09 100.68%
> reverbmp
> 3585 kargl 2 141 0 552M 208M CPU7 7 12:04 100.68%
> reverbmp
>
>
> Note, pid 3583 is the boss and I've sorted top(1) by time.
> Two observations: (1) the time accumulation for the images is
> skewed, which suggests load balancing is still an issue. (2) The
> % CPU suggests that processes can be penalized while switching
> among the cpus.
>
> Now, let's reboot with 4BSD.
>
> % sysctl -a | grep 4BSD
> kern.sched.available: 4BSD,ULE
> kern.sched.name: 4BSD
>
> % mpirun -n 8 --hostfile mf ../../build/bin/reverbmp reverbmp.in
> ...
> Wall time: 15.320 min
> CPU time: 15.320 min
>
> % mpirun -n 16 --bind-to none --hostfile mf ../../build/bin/reverbmp
> reverbmp.in
> ...
> Wall time: 13.955 min
> CPU time: 13.941 min
>
> The top(1) output is similar to the above ULE cases with
> '-n 8' and '-n 16'. That is, the workers all have 99.xx%
> CPU.
>
> % mpirun -n 17 --bind-to none --hostfile mf ../../build/bin/reverbmp
> reverbmp.in
> ....
>
> Wall time: 13.893 min
> CPU time: 13.072 min
>
> last pid: 2972; load averages: 17.09, 16.82, 14 up 0+00:46:02
> 12:31:04
> 66 processes: 18 running, 48 sleeping
> CPU: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
> Mem: 3382M Active, 87M Inact, 841M Wired, 298M Buf, 58G Free
> Swap: 16G Total, 16G Free
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU
> COMMAND
> 2951 kargl 2 52 0 552M 207M CPU8 8 11:24 94.33%
> reverbmp
> 2944 kargl 2 52 0 544M 207M CPU7 7 11:24 94.33%
> reverbmp
> 2954 kargl 2 52 0 552M 207M CPU3 3 11:24 93.53%
> reverbmp
> 2946 kargl 2 52 0 544M 207M CPU11 11 11:24 92.73%
> reverbmp
> 2958 kargl 2 52 0 544M 207M CPU10 10 11:24 91.97%
> reverbmp
> 2942 kargl 2 52 0 800M 286M RUN 6 11:24 97.48%
> reverbmp
> 2957 kargl 2 52 0 544M 207M CPU15 15 11:24 92.75%
> reverbmp
> 2948 kargl 2 52 0 544M 207M CPU5 5 11:24 99.04%
> reverbmp
> 2949 kargl 2 52 0 544M 207M CPU6 6 11:24 92.76%
> reverbmp
> 2945 kargl 2 52 0 544M 207M CPU12 12 11:24 92.76%
> reverbmp
> 2943 kargl 2 52 0 544M 207M CPU4 4 11:24 96.69%
> reverbmp
> 2947 kargl 2 52 0 544M 207M CPU1 1 11:24 95.90%
> reverbmp
> 2955 kargl 2 52 0 544M 207M CPU13 13 11:24 97.47%
> reverbmp
> 2956 kargl 2 52 0 544M 207M CPU9 9 11:24 92.04%
> reverbmp
> 2950 kargl 2 52 0 544M 207M RUN 2 11:24 95.90%
> reverbmp
> 2952 kargl 2 52 0 544M 207M CPU0 0 11:24 96.67%
> reverbmp
> 2953 kargl 2 52 0 544M 207M CPU14 14 11:24 92.73%
> reverbmp
>
> The image with lowest pid is the boss, i.e., pid 2942. The
> accumulated time and % CPU numbers suggest a round-robin like
> scheduling. More importantly, one see
>
> ULE:
> Wall time: 16.456 min
> CPU time: 15.358 min
>
> 4BSD:
> Wall time: 13.893 min
> CPU time: 13.072 min
>
> One of these is better from a HPC viewpoint.
>
Interesting.
Just showing a little more of the potential tradeoff structure visible
in the -n 16 vs. 17 example data:
ULE -n 16 .vs 17
Wall time: 13.905 min vs. 16.456 min (16.456/13.905 approx.= 1.184)
CPU time: 13.904 min vs. 15.358 min (15.358/13.904 approx.= 1.105)
Also: 16.456/15.358 approx.= 1.071
4BSD -n 16 vs. 17:
Wall time: 13.955 min vs. 13.893 min (13.893/13.955 approx.= 0.9956)
CPU time: 13.941 min vs. 13.072 min (13.072/13.941 approx.= 0.9377)
Also: 13.893/13.072 approx.= 1.063
Cross scheduler comparison/contrast ratios for -n 17 (ULE/4BSD):
Wall time: 16.456/13.893 approx.= 1.184
CPU time: 15.358/13.072 approx.= 1.175
I avoid listing ratios that are very near 1.0 by comparison.
Single run comparison/contrast can supply motivation for more complete
example data that would show variability across the attempted
reproductions. (Not necessarily appropriate for this context but might
be for after ULE is updated.)
--
===
Mark Millard
marklmi at yahoo.com