Re: Too many pythons in poudriere

From: Mark Millard via freebsd-ports <>
Date: Thu, 08 Jul 2021 16:41:13 UTC

On 2021-Jul-8, at 08:45, bob prohaska <fbsd at> wrote:

> Even with -J1 and no ALLOW_MAKE_JOBS I'm still
> seeing five pythons occupying at least 3 GB on
> the loose.

Actually I just looked and saw:

Swapinfo 7.36%

(Unlike the 83% or so I saw somewhat around 3 hours ago.)

Load Averages (220%) 2.20 2.18 1.76

Elapsed 12:54:56

I do not see a swaplog in
to look at. So I can not see how much the peak swap space
usage was so far (approximately).

> I'm fairly sure this didn't happen
> when using make by itself (IIRC it was -j2).
> I also got rid of the mistaken directive in
> poudriere.d/make.conf.

When I look at
now I see:

#.if ${.CURDIR:M*www/chromium}
#.if ${.CURDIR:M*databases/sqlite3}
#.if ${.CURDIR:M*www/firefox}

which does not match your wording.

does show ALLOW_MAKE_JOBS=yes commented out:

# By default MAKE_JOBS is disabled to allow only one process per cpu
# Use the following to allow it anyway
# List of packages that will always be allowed to use MAKE_JOBS
# regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
# which holdup the rest of the queue to build more quickly.
#ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"

(The interface for seeing the files does not show
timestamps so I can not tell when updates were done.)

(This is the wrong place for a MAKE_JOBS_NUMBER= use
but it is commented out.)

To see what is getting CPU time that leads to
the load averages being around 2 might take
using something like top sorted by cpu time
and watching for a while.

> There is a 
> in poudriere.conf, presumably GB.

Documented as GiB:

# How much memory to limit jail processes to for *each builder*
# in GiB (default: none)

Per builder, not per-make-process.
Within a builder each make-process shares
that size space with the others.

> That
> looks like a good knob to play with. Would
> setting it to something like 3 or 4 help?

If the memory use exceeds what you set, the builder
process is likely killed. It looks to be a way to
have one or both of the following true:

A) Stop the builder somewhat before the whole system
runs out of memory (and swap space). This is likely
a better failure mode handling: it could avoid the
system hanging up.

B) Prevent interfering other processes outside the
builder by being sure the system still has available
memory for other activities.

But when I looked at the scripts it turns into
jexec (and jexecd) command line options but
man jexec did not document the options that I
could see. No found for "man jexecd". So I'm

> RAM plus swap presently totals 4.6 GB.
> It looks like the present try will run out of
> swap eventually, but I'll let it go for now
> to see if anything interesting happens.

It may later fail again, but the swap space usage
is not large as of when I looked.

Mark Millard
marklmi at
( went
away in early 2018-Mar)