Python 37/38 conflict, was Re: Trubles compiling lxqt on RPi4

Sat May 15 02:29:23 UTC 2021

bob prohaska fbsd at www.zefox.net wrote on
Fri May 14 01:35:28 UTC 2021 :

> On Thu, May 13, 2021 at 01:35:50PM -0700, Mark Millard wrote:
> > You have apparently chosen to build/update ports via a
> > technique that requires you to manage the dependencies, at
> > least some of the time. (Not that when is necessarily
> > obvious up front.)
> > 
> 
> You give me too much credit 8-)
> 
> > Your environment is now based on a mix of python37 and
> > python 38 related materials. You are likely going to
> > need to rework/rebuild/reinstall things to avoid that.
> > 
> > Hints may come from that UPDATING that I quoted but
> > things are more broken overall than what UPDATING is
> > intended to cover. You might end up needing to
> > uninstall a bunch of stuff until python is unused
> > (or only one python is used) and then follow the
> > notes if you have only python37 use and want to
> > get to python38. Finally rebuild/reinstall what
> > was uninstalled, based on python38 as a context.
> > 
> Trying to deinstall both python37 and python38 didn't
> suffice. Python38's reinstall fails with the same
> conflict. Deleting the offending file doesn't help 
> If other things need to be deinstalled it's not clear
> what they might be.
>  
> > Due to how I build/install ports, I've not had to deal
> > with ending up with the mix so I'm not familiar with
> > the details for recovering from a messy mix.
> > 
> 
> Would use of poudriere help with this sort of problem?
> It isn't trivial to configure, but this sort of difficulty
> has been growing ever worse. I didn't want to deal with the 
> complexity and overhead, but maybe it's time. 
> 

I happen to use ports-mgmt/poudriere-devel and it
avoids such issues but does have build-time tradeoffs
and the like. (I'll note that I presume the sort of
sustem tuning to avoid Out Of Memory kills and I try
to scale to avoid a literal out of swap space
problems.)

I'll start with very overall background for port
building because I do not understand your context
or goals. Otherwise my material could end-up
implicitly be picking from the alternatives in
an inappropriate way. Some of this is relevant to
(all?) other forms of port building as well.

Some basic questions will be:

A) ZFS vs. UFS? (There are some configuration setting(s)
dependent on which.)

I currently have examples of both: I've started
experimenting with ZFS again in some contexts, after
years of not using it. No individual context is using
a mix of both and I use ZFS in contexts with >= 8
GiBytes of RAM. I do not try to tune it for small
memory contexts (small on ZFS's scale).


B) How a builder establishes a world-context to execute in?
For reasons of testing patches and such I build and
install a world into a directory tree and have poudriere
use that tree instead of poudriere installing or even
building its own world in a tree. (And I've never done it
any other way.)

I do this with separate world-trees for aarch64 vs. armv7
on an aarch64 system so I build for armv7 in a faster
context with more RAM and then transfer materials to
the armv7 system for pkg to use for pkg commands. (I've
not set up a server/client context.)

You could, of course, just deal with "native" and ignore
the RPi* aarch64's supporting doing armv7 builds.

I use the same buildworld for updating the running kernel
and world and for installing the world used for poudriere
when the same OS vintage/variation is to be used for both.

If you prebuild, there will be questions of what paths
you want to use to reference the for-poudriere build
trees.


C) How a builder establishes a ports tree? For reasons of
testing patches and such I have a /usr/ports tree of my
own (sometimes under another name) and have poudriere use
that tree instead of making its own.  (And I've never done
it any other way.)

I use the same /usr/ports for both aarch64 and armv7, so
only the one copy.

You might want a different path than /usr/ports if you
pre-establish the ports tree.


D) What FreeBSD versions to target? I do not happen to
use ports that must track the kernel version in detail
so I can target a releng/13's release/13.?.? and use the
ports for stable/13 as well. In fact, I can generally
get away with using those same ports on main [so: 14],
being explicit about the ABI for the pkg commands.

You might use ports to drive displays and such in a way
that leaves you required to track kernel versions more
closely. But if it is only RPi*'s, then may be not for
that. But there are other ports around that violate the
clean separation vs. kernel details.

To some extent this gets into "how many builds to cover
all the systems?". That in turn can influence how the
systems are set up, such as to eliminate some builds
being needed. Your context might be simple, with only
one type of context to cover.


E) Build as root? As non-root?

I happen to only have done build as root but the
systems are not used for other activities. There
could be ownership/permission issues that I've
not run into.


Examples of a couple of configuration files . . .

You might end up with a file like:

# more /usr/local/etc/poudriere.d/make.conf 
WANT_QT_VERBOSE_CONFIGURE=1

(I no longer remember why I enabled that.)

For a UFS context an example of the differences
with the /usr/local/etc/poudriere.conf.sample file
is as follows ( +: poudriere.conf and
-: poudriere.conf.sample ). Some of it presumes
example answers to the earlier questions.

(white space details might not go through unchanged)

# diff -u /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf

--- /usr/local/etc/poudriere.conf.sample        2021-03-09 07:43:07.000000000 -0800
+++ /usr/local/etc/poudriere.conf       2021-05-14 19:09:30.055596000 -0700
@@ -13,7 +13,7 @@
 
 ### NO ZFS
 # To not use ZFS, define NO_ZFS=yes
-#NO_ZFS=yes
+NO_ZFS=yes
 
 # root of the poudriere zfs filesystem, by default /poudriere
 # ZROOTFS=/poudriere
@@ -27,7 +27,7 @@
 # Also note that all protocols supported by fetch(1) are supported here, even
 # file:///
 # Suggested: https://download.FreeBSD.org
-FREEBSD_HOST=_PROTO_://_CHANGE_THIS_
+FREEBSD_HOST=https://download.FreeBSD.org
 
 # By default the jails have no /etc/resolv.conf, you will need to set
 # RESOLV_CONF to a file on your hosts system that will be copied to
@@ -56,7 +56,15 @@
 # yes       - Enables tmpfs(5) for wrkdir and data
 # no        - Disable use of tmpfs(5)
 # EXAMPLE: USE_TMPFS="wrkdir data"
-USE_TMPFS=yes
+#NOTE: Disable (or just "data") for < 8 GiByte RAM environments,
+#      presuming -j4 for 4 cores. wrkdir can be rather large
+#      (17+ GiByte for lang/rust) and localbase can be larger.
+#      Swap covers tmpfs space needs.
+#      For aarch64 I keep total swap somewhat under 4*RAM.
+#      For armv7 I keep total swap somewhat under 2*RAM.
+#      This avoids hard to interpret boot notices about being
+#      mistuned.
+USE_TMPFS=no
 
 # How much memory to limit tmpfs size to for *each builder* in GiB
 # (default: none)
@@ -155,6 +163,11 @@
 #
 # Example to define PARALLEL_JOBS to one single job
 # PARALLEL_JOBS=1
+#NOTE: on 2 GiByte RAM contexts I've used: PARALLEL_JOBS=2 but
+#      two llvm*'s are likely the biggest combination that
+#      could occur in my context. lang/rust or other even
+#      larger build contexts need not be appropriate.  I
+#      normally use ALLOW_MAKE_JOBS=yes .
 
 # How many jobs should be used for preparing the build? These tend to
 # be more IO bound and may be worth tweaking. Default: PARALLEL_JOBS * 1.25
@@ -188,12 +201,13 @@
 
 # By default MAKE_JOBS is disabled to allow only one process per cpu
 # Use the following to allow it anyway
-# ALLOW_MAKE_JOBS=yes
+ALLOW_MAKE_JOBS=yes
 
 # List of packages that will always be allowed to use MAKE_JOBS
 # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
 # which holdup the rest of the queue to build more quickly.
 #ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"
+ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py* gcc* llvm* rust* ghc* *webkit* *office* chromium* iridium* mongodb*"
 
 # Timestamp every line of build logs
 # Default: no
@@ -208,11 +222,22 @@
 # This defines the max time (in seconds) that a command may run for a build
 # before it is killed for taking too long. Default: 86400
 #MAX_EXECUTION_TIME=86400
+# Cortex-A53 and such are slow for the purpose, allow 4 times the defaults:
+MAX_EXECUTION_TIME=432000
 
 # This defines the time (in seconds) before a command is considered to
 # be in a runaway state for having no output on stdout. Default: 7200
 #NOHANG_TIME=7200
+# Cortex-A53 and such are slow for the purpose, allow 4 times the defaults:
+# Also: boost-libs seems to have a long time with no output but it making
+# progress in parallel builds.
+NOHANG_TIME=28800
 
+# Cortex-A53 and such are slow for the purpose, allow 4 times the defaults:
+MAX_EXECUTION_TIME_EXTRACT=14400
+MAX_EXECUTION_TIME_INSTALL=14400
+MAX_EXECUTION_TIME_PACKAGE=28800
+MAX_EXECUTION_TIME_DEINSTALL=14400
 
 # The repository is updated atomically if set yes. This leaves the
 # repository untouched until the build completes. This involves using
@@ -260,7 +285,7 @@
 # set.  Note that to use ccache with BUILD_AS_NON_ROOT you will need to
 # use a non-shared CCACHE_DIR that is only built by PORTBUILD_USER and chowned
 # to that user.  Then set CCACHE_DIR_NON_ROOT_SAFE to yes.
-#BUILD_AS_NON_ROOT=no
+BUILD_AS_NON_ROOT=no
 
 # Define to the username to build as when BUILD_AS_NON_ROOT is yes.
 # Default: nobody (uid PORTBUILD_UID)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

