Re: Too many pythons in poudriere

On 2021-Jul-10, at 10:40, bob prohaska <fbsd@www.zefox.net> wrote:
> 
> On Fri, Jul 09, 2021 at 05:12:57PM -0700, Mark Millard wrote:
>> On 2021-Jul-9, at 16:07, bob prohaska <fbsd at www.zefox.net> wrote:
> [big snip] 
> 
> 
> 
>>> Just to be clear, after updating kernel and world I gather the 
>>> suggestion is to repeat
>>> 
>>> # cd /usr/src
>>> # make installworld DESTDIR=/usr/local/poudriere/poudriere-system DB_FROM_SRC=1
>>> # make distrib-dirs DESTDIR=/usr/local/poudriere/poudriere-system DB_FROM_SRC=1
>>> # make distribution DESTDIR=/usr/local/poudriere/poudriere-system DB_FROM_SRC=1
>> 
>> I though you had already synchronized /usr/local/poudriere/poudriere-system
>> to match /usr/src/ and the host root file system in use. If so, no reason
>> to repeat until /usr/src/ (and the booted host root file system) is updated
>> for some reason. Avoid mixes where /usr/src/ is mismatched with the root
>> file system at the time.
>> 
>> Of course, you might want to do another update to those for all I know.
>> 
> 
> I tend to update /usr/src and /usr/ports somewhat randomly, based on
> elapsed time or in response to trouble of some kind. It looks like that
> habit may be the source of at least some of my trouble. 
> 
>> One thing you did not show was the command for after the above block of
>> 4 commands for updates:
>> 
>> # make -DBATCH_DELETE_OLD_FILES delete-old DESTDIR=/usr/local/poudriere/poudriere-system DB_FROM_SRC=1
>> 
>> If you have not done that yet, it would be appropriate.
>> 
>> For the current context of starting over, another command that
>> is appropriate (that would normally be delayed to after all the
>> poudreire bulk commands are done [were prior package builds
>> were instead being put to use]) is:
>> 
>> # make -DBATCH_DELETE_OLD_FILES delete-old-libs DESTDIR=/usr/local/poudriere/poudriere-system DB_FROM_SRC=1
>> 
> I got burned by a delete-old-something long ago, broke most of the installed ports, and haven't
> touched it since. In this case it does seem appropriate. Added to the update to-do
> list.

delete-old-libs should only be used after the ports/packages
have been updated and installed --unless one is starting from
scratch anyway.

>> I'll note that the host updating procedure also involves using
>> delete-old and, after package updates are all installed,
>> delete-old-libs . (No explicit DESTDIR= or DB_FROM_SRC= use
>> involved for the root file system update.)
>> 
> 
> It resembles the command that bit me....

My guess is that you ran delete-old-libs before having
installed updated packages/ports.

Having old libraries can lead to its own problems if they
continue to be used in places instead of those places
updating to the newer library versions.

>>> Next, 
>>> cd /usr/local/poudriere
>>> poudriere jail -d main -C all
>>> to get rid of the old jail and packages, then re-make the jail with 
>>> 
>>> # poudriere ports -c -m null -M /usr/ports
>>> # poudriere jail -c -j main -m null -M /usr/local/poudriere/poudriere-system -S /usr/src -v 14.0-CURRENT
>>> 
>>> Have I got this right? One thing I'm hesitant about is 
>>> the -C all option. 
>> 
>> No reason to delete and recreate the jail, just do a bulk build that
>> rebuilds the ports. The first "poudriere bulk" of a sequence can use
>> the "-c" option to clear out the existing packages (and logs) before
>> the builders start building.
>> 
> 
> Aye, there's the rub.... It appears that incremental updates are a
> costly approach (world/kernel, then a port, then world/kernel...).
> It seemed one could get away with it using make in the ports tree,
> though perhaps some of my troubles with that approach had the same
> origin.  

You are still trying to get back to a correct environment so
things are not normal yet. Plus you are trying to build things
that are too big for the environment you are using to just
automatically handle it. So you are having to manage the
resource use explicitly as you go.

[I do not do that: so far I've used systems that have enough
resources to allow ALLOW_MAKE_JOBS= and all cpus(/cores) to
be used by each builder for the things I build, devel/llvm*'s
tending to be the largest builds but two or more of those
at a time has happened. I'd likely need adjustments for
whatever the biggest port builds are.]

Use of -c is a very rare thing for me. But when your /usr/src/
goes from something like 1400024 to 1400025, poudriere should
notice and to a full rebuild automatically. (So the in world
installed in /usr/local/poudriere/poudriere-system better be
a match to the /usr/src/ at the time.)

The reason for the full rebuild is that the type of change
might require it. There is no separate, more detailed,
indication about what actually needs to be built.

Anyway, poudriere will fully rebuild on a regular basis
during periods where that number is changing on a regular
basis. main tends to have such changes on a fairly regular
basis.

>> As far as I know you eventually want to install:
>> 
>> # pkg install chromium
>> 
>> (that will also install things required to run chromium).
>> 
>> I'm not sure if you have a list of other packages that
>> you want explicitly installed. So my notes below need
>> not cover everything that you want to do.
>> 
> 
> Getting a running version of chromium was the origninal goal.
> The train seems to have run off the tracks......8-)
> 
>> (I do not know if ALLOW_MAKE_JOBS= and such is appropriate for the
>> llvm10 bulk build but it may fit in the resources okay:)
>> 
>> # poudriere bulk -j main -c devel/llvm10 > bulk_from_scratch_for_llvm10_1st.log
>> 
>> (I'm unclear if you would want a -J? to limit the builders
>> count for the above command: if so, use it too.)
>> 
>> That -c will first clear out the existing package builds (and
>> the logs according to the documentation). Then it will start
>> from scratch building the required ports.
>> 
>> This will build a lot of ports, those needed to in turn build
>> devel/llvm10 . So having -J1 used and ALLOW_MAKE_JOBS missing
>> would make for a very long build time. You probably want to
>> allow some combination of multiple cores to be in use,
>> possibly also involving MAKE_JOBS_NUMBER= use. You know more
>> about what combinations worked before.
>> 
>> Reminder that ALLOW_MAKE_JOBS= control would go in:
>> 
>> /usr/local/etc/poudriere.conf
>> 
>> but MAKE_JOBS_NUMBER= control would go in:
>> 
>> /usr/local/etc/poudriere.d/make.conf
>> 
> 
> That is a helpful distinction unclear to me till now.
> 
>> (I do not know if ALLOW_MAKE_JOBS= and such is appropriate for the
>> rust bulk build but some combination may fit in the resources okay:)
>> 
>> # poudriere bulk -j main lang/rust > bulk_for_rust_2nd.log
>> 
>> (I'm unclear if you would want a -J? to limit the builders
>> count for the above command: if so, use it too. You know
>> more about what combinations worked before.)
>> 
>> (Without ALLOW_MAKE_JOBS= is likely appropriate --or at least the
>> MAKE_JOBS_NUMBER= may be needed-- for:)
>> 
>> # poudriere bulk -j main www/chromium > bulk_for_chromium_3rd.log
>> 
>> (I'm unclear if you would want a -J? to limit the builders
>> count for the above command: if so, use it too. You know
>> more about what combinations worked before.)
>> 
>> Note that the management of the limited RPi3B context involves
>> adjusting things like ALLOW_MAKE_JOBS= and -J? usage for each
>> huge build to limit resource use to an appropriate scale for
>> those builds. For smaller builds, more can be allowed (more
>> adjustments).
> 
> Since I usually play with ports one at a time, is there any
> advantage to using testport versus bulk? Better diagnostics,
> maybe? 

man poudriere-testport:

     The specified port will be tested for build and packaging problems.  All
     missing dependencies will first be built in parallel.  TRYBROKEN=yes is
     automatically defined in the environment to test ports marked as BROKEN.
     See FLAVORS in poudriere(8) for supported FLAVORS syntax.\

but has a longer description in man poudriere. I'm going to
quote that about what I think is different from bulk vs. the
same as bulk when a single port is listed on the poudriere
command line:

QUOTE [with notes]
   Test a single port
     This second example show how to use poudriere for a single port.  Take
     the example of building a single port;

           poudriere testport -o category/port -j myjail

     all the tests will be done in myjail.

     It starts the jail, then mount the ports tree (nullfs), then mounts the
     package dir (poudriere/data/packages/<jailname>-<tree>-<setname>), then
     it mounts the ~/ports-cvs/mybeautifulporttotest (nullfs) it builds all
     the dependencies (except runtime ones) and log it to
     poudriere/data/logs/testport/jailname/default/mybeautifulporttotest.log).

[Ignore the ~/ports-cvs/mybeautifulporttotest detail from what I can tell.
However, I believe this is indicating that the places that some things would
go for normal use will not be used. Some temporary place is apparently
used instead and the material in that temporary place is not kept long term.]

     If packages for the dependencies already exist, then poudriere will use
     them.

[Other than a temporary spot vs. the normal place, this is like bulk.]

     When all the dependencies are built, packages for them are created so
     that next time it will be faster.

[This does indicate that, like bulk, the dependencies will be remembered
and reused when built. That does not say that the port being tested will
be remembered or resued, however.]

     All the dependency phase is done with PREFIX == LOCALBASE.

     After that it will build the port itself with LOCALBASE != PREFIX

[There is a testport option -P for "Use custom prefix".]

[bulk would use the normal PREFIX == LOCALBASE unless the port
itself makes assignments. This is a test for if the port is handling
PREFIX and LOCALBASE correctly. The earlier
~/ports-cvs/mybeautifulporttotest material is possibly
trying to reference to where the unusual place PREFIX might be.
But I see nothing in the scripts constructing that kind of path
automatically.]

     and log
     the build to
     poudriere/data/logs/testport/jailname/default/mybeautifulporttotest.log

[Again ignore the "mybeautifulporttotest.log" naming detail.
bulk would not put the log file under
poudriere/data/logs/testport/jailname/default/ .]

     Poudriere will try to: install it, create a package from it, deinstall
     it, check for cruft left behind and propose the line to add to pkg-plist
     if needed.

[bulk would not do such an install/create package from installed/deinstall
in some temporary place to test the installation/deinstallation handling.]

END QUOTE

I do not see any differences there that helps in your build
activities or for better error/warning messages about the
types of errors that you get. I'd just use bulk, listing one
port on the command line, not testport . (I do not use
testport because I'm not developing ports or significant
changes to port Makefiles and such.)

Using testport does not change needing to manage the
resource use for devel/llvm10 lang/rust www/chromium
steps on the RPi3B.

Nothing will get rid of having to do that management
for building on a RPi3B.

The only thing that will get rid of needing to the that
management is to use something other than a RPi3B that
has more resources (especially RAM in proportion to
core counts).

>> I do not know if there are other ports that you would want
>> to build and later install that are not automatically built
>> by the above or if you have a list of such in a file that
>> can be used with -f . In your context this would probably
>> be a list of ports that do not involve huge builds, given
>> that you had already done the above bulk runs (and any others
>> that might involve more huge builds).
>> 
> 
> It's clear that by segregating build dependencies from run
> dependencies poudriere provides a valuable service. Might
> there be a make target in ports that cleans up no-longer-needed
> build dependencies during installation of a given port? That
> would, I think, have been enough to avoid the python27 conflicts.

How many other ports might have the same build dependency?

How many other ports might have a run dependency on the same
port that is just a build dependency of the port in question?

How many other ports would be damaged by the removal of what
the specific port has as only a build dependency?

Rebuilding each dependency each time a port is built would
use massive time and resources (fully independent builds).
poudriere manages to reuse built ports in more contexts
instead of rebuilding them but keeps things clean anyway.

Dependecy management is not a local-to-port thing but a
more global thing, unfortunately.

After poudriere has everything built for later installation:

# pkg delete -a
. . .
# pkg install chromium
. . .

will remove all the ports (except pkg) and then install
just the ones needed for chromium's operation.

> Is there a plain-language description of the logic poudriere follows
> for building ports? I found this:
> https://forums.freebsd.org/threads/pkg-package-repository-using-ports-mgmt-poudriere-with-or-without-zfs.38859/
> but it doesn't really touch on the topics I'm looking for. Something
> on the level of a simplified flowchart or (gasp!) PowerPoint, maybe? 

Out of all the things involved in poudriere's operation
I've no clue how to identify just the possibly small
subset you may be curious about. I doubt that you want
to learn about all aspects of poudriere's operation.
(Well, beyond my range of knowledge. I investigate as
needed as I go along.)

What you sight is (old) how-to-use documentation, not
how-does-it-work documentation. So I'm not even sure
which you are after.

Either way, I'm not aware of anything that is a likely
match to whatever you are hoping to see. A lot of your
questions are of the nature "how do I side step the
available commands for how to use poudriere".

"Plain-language" and jails and repositories with
transitional updates (all updates completed vs.
no updates being the two results, no mixed results)
and such do not tend to go together.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)