Large builds with poudriere

[Mark Millard]

On 2021-May-20, at 11:11, Mark Millard <marklmi at yahoo.com> wrote:

> Andrea Venturoli ml at netfence.it wrote on
> Thu May 20 14:54:50 UTC 2021 :
> 
>> On 5/20/21 4:11 PM, Alexis Praga wrote:
>>> 
>>> Thanks for the suggestions.
>>> 
>>>> At least, if your machine is powerful enough, you can edit your
>>>> /usr/local/etc/poudriere.conf and list these big ports in
>>>> ALLOW_MAKE_JOBS_PACKAGES.
>>> I'm using ALLOW_MAKE_JOBS=yes already so it won't help unfortunately, I think.
>> 
>> Those are not the same things. Check the docs.
>> 
>> Briefly, ALLOW_MAKE_JOBS will allow building several packages at once 
>> (one per core) with one core dedicated to each.
> 
> Nope: /usr/local/etc/poudriere.conf.sample reports that it enables
> more than one process per cpu:
> 
> # By default MAKE_JOBS is disabled to allow only one process per cpu
> # Use the following to allow it anyway   
> # ALLOW_MAKE_JOBS=yes

My wording was poor by being incomplete: ALLOW_MAKE_JOBS allows each
builder to have as many processes as there are cores/processors. So if
multiple builders are allowed (and happen to be in use), then the total
number of processes ready to run for building can be bigger than the
number of cores/processors, something that shows up in the load averages.

How many builders are allowed can be controlled with:

# parallel build support.
#         
# By default poudriere uses hw.ncpu to determine the number of builders.
# You can override this default by changing PARALLEL_JOBS here, or
# by specifying the -J flag to bulk/testport.
# 
# Example to define PARALLEL_JOBS to one single job
# PARALLEL_JOBS=1

Looking in /usr/ports/Mk/bsd.port.mk there is also the following,
that is part of where some of the "MAKE_JOBS" terminology comes
from:

# MAKE_JOBS_UNSAFE
#                               - Disallow multiple jobs even when user set a global override.
#                                 To be used with known bad ports.
# DISABLE_MAKE_JOBS
#                               - Set to disable the multiple jobs feature.  User settable.
# MAKE_JOBS_NUMBER
#                               - Override the number of make jobs to be used.  User settable.
# MAKE_JOBS_NUMBER_LIMIT
#                               - Set a limit for maximum number of make jobs allowed to be
#                                 used.

>> ALLOW_MAKE_JOBS_PACKAGES will let a single package build using more cores.
> 
> 
> Even when MAKE_JOBS is otherwise disabled:
> 
> # List of packages that will always be allowed to use MAKE_JOBS      
> # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports 
> # which holdup the rest of the queue to build more quickly.
> #ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)