Python 2.7 removal outline

[Olivier Certner]

Hi,

Maintainer of Tauthon here, and of Pale Moon (for the few hours it lived in 
the tree in February; but I'm still pushing updates to PR 251117).

I find this announcement very much disappointing, because the situation for 
ports that need Python 2.7 or similar to build doesn't seem to have changed at 
all. In short, we are just told (again) that they should disappear.

But now we are told also that there are exceptions to the general rule. And 
even worse (or better, for the comical effect), that the Chromium exception is 
going to force its agenda on portmgr@, because nobody seems to really know 
when that migration will complete (having quickly studied how it worked out 
for Firefox, I'm willing to bet this will never happen before June, and likely 
will not before the end of this year either). "I love it when a plan comes 
together."

I had hoped that portmgr@ would turn to me (and others in the same situation) 
with at least some way out to allow Pale Moon to go into the ports tree. In 
its mail, the closest thing I can find is this:
> - you are of course free to provide your own version of Python 2.7, Tauthon
>   and any application using those languages in your local setup, by using
>   overlays for example.
So, if I understand correctly, in its great magnanimity, portmgr@ allows us to 
do what we want locally. Gosh! That's news. Didn't know I had been violating 
the law for the past 10 years by doing exactly that with my set of patches to 
the ports tree (a long way before overlays even existed; that said, I think 
overlays are helpful, more on that below).

And there's more in the same "tone":
> No excuses. * 3
You probably were not addressing this to me in particular (at least I hope). 
But just in case, did you mean: Excuses for not respecting a non-existing 
policy?

In December last year, I submitted Tauthon with the goals to be able to use it 
to build Pale Moon and help others that needed to run 2.7 code despite phasing 
out of PSF's interpreter. It was pushed on 2020/12/11. I then submitted an 
upstream-approved official version of Pale Moon port, that was pushed on 
2021/02/15. It was removed less than 4 hours later, together with Tauthon, in 
"rage commit" 565350 by antoine@, without any public (or private to me) 
communication, before or after. Except for one thing: He responsed to my 
request for explanations by saying: "When we deprecate python 2.7, we also 
deprecate all forks of python 2.7.".

That also was news to me. Don't know how many ports would have to be removed 
if this rule was followed for all forks of now deprecated projects. I don't 
think there would be only a few of them. What about the fact that Tauthon has 
lived 2 months in the tree up to the removal? What about that it was 
contributed through a public PR, linked to other Python-related PRs by some 
committers? What about that nobody issued a public objection to its presence 
on said PR or on mailing lists? What about that it is still listed *today* on 
the wiki's WantedPorts page? Never had a single answer on all this. Surely 
these must have been signs that what I was doing was very wrong indeed, and 
against a very clear and strong policy, which I should have understood by 
myself. Idiot me.

I'm not going to repeat here the long mail I wrote in response to FUD being 
spread about Pale Moon (and even Tauthon and Python 2.7 to some extent) by 
rene@ and danfe@, and its excruciating details on how deluded they were 
(still are?). These mails were for committers only, and exchanged on 
2021/02/2{1,2}.

I'll just quickly point out the new FUD and inaccuracies, add some facts and 
opinions to the picture, and wrap up with two possible ways out.

> Tauthon is not guaranteed to be compatible with any official
>   Python version so keeping it would just unnecessarily complicate things.

mandree@ preceeded me, so not going to add much except that Tauthon in 
practice lives up to its promise (there are incompatibilities due to the 
introduction of some Python 3 features, but they are very minimal and hard to 
trigger; unfortunately for me, Pale Moon's weird build system, inherited from 
Mozilla, triggered some).

As to why it would complicate things, we are left with no clue, especially 
given that Tauthon is not hooked at all into python.mk and that no port 
currently depends on it (Pale Moon having been removed).

> - On a related note, most software using Python 2.7 was already removed
>   from the Ports Tree last year, a lot of it being unmaintained or
>   more or less abandoned upstream.

1. Most is not all, unfortunately. What happens for the rest? Is "Well, let's 
pretend it doesn't exist, and shut the contributors up." your response? 
Seriously?
2. Security is a non-issue for build-only dependencies.

> - We are indeed faster with dropping Python 2.7 than e.g. Ubuntu, however
>   more recent Debian/Ubuntu distributions are more and more dropping Python
>   2.7 too. This also has to do with how their branching model works, the
>   package set of Ubuntu LTS is determined a few months before the release
>   itself.

Debian is still tolerating Python 2.7 for build-only dependencies in bullseye, 
which is due to be released imminently, and will be supported until around 
2024. Ubuntu 20.04 LTS incorporates it, apparently without restrictions (I see 
a full suite of packages relying on 2.7 there), and this release will be 
supported until... April 2025. So, yes, faster by at least 2 years.

Surely, we are not organized the same, and do not have the same manpower and/
or money. However, their security teams do not seem to think that phasing out 
CPython 2.7 right now is of uttermost importance. Some Debian links on the 
topic:
https://tracker.debian.org/pkg/python2.7
https://wiki.debian.org/Python/2Removal
I must point out that this last page, although listing interesting links, 
seems itself seriously outdated, as it is contradicted by facts (e.g., 2.7 is 
in bullseye, and it is indeed receiving security fixes, see the first link). 
It seems that they have changed their mind in light of needs and demands. Food 
for thought for portmgr@?

And again, there would be no hurry at all for build-only dependencies. Or is 
there? May I ask on which ground exactly?

> As can be seen on [2], multiple vulnerabilities already have
> been fixed for Python 3.6 to 3.9 this year.
> [2] https://www.python.org/downloads/release/python-392/

I've started looking into these vulnerabilities. Most are simple to 
understand, and their patches even readily apply to Tauthon when relevant. 
Going to submit a bunch of them upstream. At least, this is possible with 
Tauthon, contrary to CPython 2.7.

But in the end, I don't think this is really important for the dependent ports 
issue, since, again, we are talking about build-only dependencies on CPython.

That was just for the sake of re-establishing a more accurate balance of 
facts. Given the track record of recent reactions of portmgr@, I'm now not 
foolish enough to believe that all that precedes is going to have any visible 
effect on them.

Now, for the two possible ways out, I'm still having some hope (but frankly 
not that much).

1. Add the infrastructure to have build-only dependencies. I've proposed 
changes to that end (https://reviews.freebsd.org/D28946). In addition to the 
comments in the review, bapt@ rightfully pointed out that 'make install' would 
still be possible to run for ports listed in NORUNTIME. I acknowledge that 
this is indeed a problem in the current problem, but think it could be solved 
technically (e.g., forbidding 'make install' for those ports, but allowing it 
when building a dependent port through an environment variable, and removing 
the install after the build). Which reinforces my thinking that the "problem", 
whatever that is, is not technical, but human. Overall, portmgr@ doesn't 
really seem to be interested in this solution (got short reactions such as 
"with RESTRICTED, we don't need this", or "this would be a precedent", indeed 
a useful one if you're asking me).

2. Leverage overlays to provide additional repos, a bit like AUR for Arch. 
Here I'm in fact building on top of one of bapt@'s ideas. Sounds great for 
publishing ports that are not in the official tree. But not necessarily for 
package building: I personally won't commit to maintaining a separate build 
cluster for all arches and supported FreeBSD versions, in the short term at 
least.

I re-read portmgr@'s charter (https://www.freebsd.org/portmgr/charter/). I 
wish it contained points about proper planning, communication and helping 
maintainers and committers instead of destroying their work without notice, 
even for "niche" ports. Perhaps it doesn't because this was implicit or taken 
for granted. In which case, in light of recent events, it may be a good time 
to revise it.

-- 
Olivier Certner