Re: Restraining poudriere

From: Jose Quinteiro <freebsd_at_quinteiro.org>
Date: Sat, 12 Jun 2021 13:26:16 -0700
On 6/12/21 10:57 AM, bob prohaska wrote:
> On Sat, Jun 12, 2021 at 07:36:48PM +0200, Michael Gmelin wrote:
>>
>>
>>> On 12. Jun 2021, at 19:31, bob prohaska <fbsd_at_www.zefox.net> wrote:
>>>
>>> ???In playing with poudriere on raspberry pi 3 and 4 it seems to
>>> work well on the 8 GB Pi4 but is over-optimistic on the 1 GB Pi3.
>>>
>>> Can poudriere be configured to tackle packages one at a time?
>>
>> Yes, see poudriere.conf:
>>
>> # parallel build support.
>> #
>> # By default poudriere uses hw.ncpu to determine the number of builders.
>> # You can override this default by changing PARALLEL_JOBS here, or
>> # by specifying the -J flag to bulk/testport.
>> #
>> # Example to define PARALLEL_JOBS to one single job
>> # PARALLEL_JOBS=1
>>
>> -m
>>
> 
> I perhaps misunderstood what was meant by "builders", confusing it
> with threads. Or maybe cores....
> 
> Trying it now, hoping to see parallel core use.

You won't. Setting PARALLEL_JOBS=1 means exactly one Poudriere worker
will run, and that make will not build in parallel. You have to decide
whether you want multiple Poudriere jobs, each using a single core at
most, one Poudriere job with make(1) potentially using several cores, or
some combination of both.

The three variables that control this are:

PARALLEL_JOBS, as you already know. This is the maximum number of
workers Poudriere will launch. It can be less than this if there are
multiple dependent packages waiting for a big package to build. This
happens a lot with Llvm and the like. Not having ALLOW_MAKE_JOBS set is
very frustrating in this case, because the big pig of a build will use
one core at most and take forever.

ALLOW_MAKE_JOBS and MAKE_JOBS_NUMBER (make.conf). The first one allows
make(1) to launch parallel jobs, and the second controls the maximum
number of make jobs that will be launched. The problem here is that many
builds have lengthy steps that are not concurrent (I'm looking at you,
autotools!) A lot of times the actual compile and link steps take a tiny
fraction of the total build time.

I chose to use a combination of both, and had to experiment with
different numbers for PARALLEL_JOBS mad MAKE_JOBS_NUMBER so that a
majority of my cores were used for the build, but I did not run out of
memory:
https://forums.FreeBSD.org/threads/poudriere-never-completes-first-run-without-errors.78000/post-486707

> 
> Thanks for writing!
> 
> bob prohaska
> 
You're welcome!
Jose
Received on Sat Jun 12 2021 - 20:26:16 UTC

Original text of this message