Python 37/38 conflict, was Re: Trubles compiling lxqt on RPi4

Mark Millard marklmi at
Tue May 18 02:49:25 UTC 2021

bob prohaska fbsd at wrote on
Mon May 17 23:46:38 UTC 2021 :

> On Mon, May 17, 2021 at 12:28:24PM -0700, Mark Millard via freebsd-ports wrote:
> > bob prohaska fbsd at wrote on
> > Mon May 17 15:55:21 UTC 2021 :
> > 
> > > The existing conflict between versions of python strikes me as more of a 
> > > planning problem than a software bug. It may be naive, but I don't see
> > > why python37 and python38 can't use distinct names for files placed in
> > > system directories.
> > 
> > You seem to be under the impression that absent having
> > any common file paths, things would just work. This
> > seems unlikely. A simplified presentation of why
> > follows.
> > 
> I'm under the impression that absence of common paths would help
> reduce conflicts.

True: possibly necessary, even if not sufficient.

> In the case at hand it might be sufficient. 

I do not know: I do not have a very complete understanding
of the full status of your environment after the problem.
(Nor of just what specifically lead to the problem.) I do
know that I do not deal with the issue in my context --but
that is because I use procedures that avoid the general
type of issue (tolerating any other tradeoffs involved).
(I have worked via portmaster and just plain make in the
past before using poudriere as well.)

> > Imagine two programs:
> > 
> > p37 that is set up for python37
> > p38 that is set up for python38
> > 
> > imagine that both use a library plib that is
> > not internal to each but external to both.
> > 
> > So should plib be built/present for python37?
> > python38? Both?
> > 
> > If both: it suggests building a huge set of python
> > software multiple times, once per supported version
> > in the range of to-be-supported python versions. (I
> > do assume python versions make for some degree of
> > incompatibility distinctions. It might be only only
> > some version changes have that property. But such
> > would not change the basic point.)
> > 
> It suggests that p37 (installed first) would install
> its preferred version of plib. When p38 is installed, it
> would test for a compatible version of plib

So now p38 is required to classify all prior combinations
of versions of external libraries it might use (such plib),
and to put in tests for handling all the combinations.

And what if p38 is installed first and p37 second? p37
must do similarly --but has no way to well classify
combinations involving library versions that did not
exist when it was released.

One alternative in the industry is having each package
fully self contained (up to the interfacong with the OS
or whatever the kind of context is). The package-build
builds everything required and bundles what is needed
at run time all together so the package does not depend
on any other packages: its installer and installation
results are self sufficient (up to the "OS"). This
makes other tradeoffs.

There are examples that are similar in ports, some
tied to bootstrapping issues. For example, building
ports-mgmt/pkg builds far more internally because
pkg can not depend on other ports/packages that would
need pkg to already be operational to get things


As I understand lang/rust contains and builds its
own (subset of?) some llvm version instead of
depending on one of llvm10/llvm11/llvm12. Its
build time and resource use may be illustrations
of some of the tradeoffs in this style: its build
of an llvm does no good at saving time for any
other port build that involves the same vintage of

> and add a new 
> one, with a different name, if the prior version isn't 
> satisfactory. Libraries already seem to have a variety of
> suffixes on their names, so it appears something of the sort
> is already going on. Am I completely missing the point?

See notes above.

> > I know, for example, python39 invalidated code in
> > something I've built in a non-FreeBSD context. The
> > software had to be modified to be compatible with
> > both older python's and python39 (if they wanted
> > to support the older versions as well going
> > forward). (It was not a context of wanting to take
> > advantage of new things in the newer python. But
> > that kind of context is not universal.)
> Not sure I see a fundamental problem here. Python
> 38 remaining useful/necessary after introduction of
> 39 doesn't seem so bad. It seems the norm.

How far back are the pre-supplied older versions
supposed to go? On operating system A? B? C? (Likely
differing choices will be made.) How many old
versions continue to get bug fixes and security
updates and the like? How much less effort is put
into creating new versions (supposed improvements)
as a consequence?

To again use p37 and p38: how do they deal with the
varying range of old versions on A vs. B vs. C?

Some of this is or can be done, but the extent tends
to be rather limited because the information
requirements and effort to manage much of a variety
is huge. And it does not indefinitely delay having
to deal with updating to get things working --unless
the history covered always goes back to the beginning.
Otherwise, it just changes some about when the issue
is faced.

> > Most ports having a separate upstream to deal with
> > and having a huge number of ports makes for
> > port-developer and upstream-developer coordination
> > based solutions having great difficulties overall.
> >
> Indeed, and they're getting worse over time.
> > No technical-content has been presented to make these
> > sorts of problems disappear. With the problems being
> > present, it is a matter of working in a way that
> > avoids running into the problems or with dealing with
> > fixing things when the problems occur. 
> I've wondered from time to time if it's possible, even
> only in principle, to make the entire ports tree build
> in one invocation.

It is called "poudriere bulk -a . . ." --as used by
some of the FreeBSD build server port builds if I
understand right. Over 30,000 ports when no prior
incomplete run's partial results are around.

I once did a "poudriere bulk -a" when the ThreadRipper
1950X was new and I was testing it. (At the time FreeBSD
had been having problems with the new type of AMD
processors and I was looking to see what all would fail.
Then I continued, only dealing with the failures.)

The ThreadRipper 1950X is the only machine I've had
access to that I'd ever try such a build with. Otherwise
my time preferences would not allow waiting sufficiently
long. (It is possible no other machine had sufficient
resources to complete the process in a time frame I
would tolerate, such as storage-space/RAM+swapspace

> At the moment, the answer seems to be 
> "no". But is it "no" on first principles?

The answer is yes: "poudriere bulk -a" is an
example way of doing so.

I'm not sure if anyone has tried a full build of
all the ports tree without using poudriere to do
it. At this point, poudriere, or something somewhat
analogous to it, may be the only kind of port-build
context that has a chance of completing the process

In that sense, poudriere is the most extensively
tested of the ways of building the ports: fairly
regular use to build the complete ports tree in
one go.

> > I've done both
> > basic styles over the years and recognize tradeoffs.
> > I'm happy to help someone explore one of the ways
> > in which poudriere can be an alternative. It is not
> > for me to declare how well it would end up fitting
> > their goals, context, preferences, and so on vs.
> > other alternatives overall.
> >
> I'll  continue to explore poudriere.  Your efforts are 
> much appreciated, but also rather daunting.

Any particular questions still pending? Non-obvious
steps for your context that you wonder about? Have
you been able to build some port via poudriere yet?

> The learning
> curve is steep

I only managed my first poudriere experiments when someone
did something similar for me, reporting what they changed
from the basic install (and so implicitly: what they left
alone). It allowed me to fairly well isolate what I needed
to deal with explicitly to get started, given my goals
and odd context (mostly tied to PowerMac support at the

> and resource requirements are high.

An example of: Using poudriere has tradeoffs involved.

I do not know what using spinning rust would be like
for how long things take.

My contexts generally have had a USB3 SSD as the storage
media --or better in some cases. I've never tried this
with spinning rust or microsd cards or networked file
systems. (The PowerMacs had SATA SSDs, but not modern
SATA, so possibly not "better". But still not spinning
rust nor microsd cards.)

Ignoring old PowerMacs, possibly the slowest context
was armv7 (Cortex-A7) with 1 or 2 GiBytes of RAM, 4
cores, and use of the USB3-capable SSD type of
storage device on a USB2 bus instead of USB3 (in some
cases with a powered hub needing to be involved). I
tend to build for armv7 on aarch64 using an armv7
installworld that poudriere is configured to use for
one of its jails.

The USB3-capable SSDs are 240 GiByte ones. Only on
old PowerMacs have I ever used anything with less
capacity (120 GiByte) for this kind of activity.
I do have access to some examples of higher capacity
and faster media than SATA SSDs.

After the 2-sockets/2-cores-each PowerMacs both
had effectively died (over heating), I greatly
restricted what I build (now on the 2 core PowerMacs).
I still build for 32-bit powerpc on the powerpc64,
much like the armv7 on aarch64 case. (Other issues
have meant that the PowerMacs are not in regular
use, however.)

> If 
> there's some larger point I'm missing please give a hint.

Mark Millard
marklmi at
( went
away in early 2018-Mar)

More information about the freebsd-ports mailing list