portmaster new development

[Stefan Esser]

Am 27.12.20 um 06:04 schrieb Thomas Mueller:
>>> And as portsnap user I have a question: Do they planning deprecation of
> 
>>> portmaster too?
> 
>> No, I'm actively working on portmaster and have rewritten it from
>> scratch for better performance (and additional features, e.g. building
>> in a clean chroot jail, similar to synth or poudriere).
> 
>> I have been using that version for more than one year, but the
>> functionality is not complete, yet.
> 
>> On a test system with > 2200 installed ports it takes less than 10
>> seconds to identify the ~600 out-of-date ports (that I keep in this
>> state for testing of the upgrade strategy function), which is more
>> than 30 times faster than the same operation with the "official"
>> portmaster.
> 
>> Until completion of that version, I'll continue to maintain and
>> update the current portmaster port ...
> 
>> Regards, STefan
> 
> Question about the relation of portsnap and portmaster reminds me of Java and Javascript, or potato and sweet potato (not closely related).

Yes, portsnap and portmaster do not share anything beyond the first 4 
letters of their names ...

> Since my question is about a new portmaster, I rename the subject to "portmaster" or "portmaster new development", rather than hijack the "portsnap" thread.
> 
> Which portmaster do I get if I build and install what is currently in the ports tree?

This is a version that I became maintainer of when it lacked flavor 
support and the original developer (Doug Barton) had left the project.

> amelia2# ls -l ports-mgmt/portmaster
> total 16
> -rw-r--r--  1 root  wheel  1479 Dec 27 02:01 Makefile
> -rw-r--r--  1 root  wheel   184 Feb 28  2018 distinfo
> drwxr-xr-x  2 root  wheel   512 Dec 27 02:01 files
> -rw-r--r--  1 root  wheel  1189 May  6  2019 pkg-descr
> 
> from a fresh svn up of the ports tree.

Yes, I added a feature requested by Kevin Obermann, yesterday, who
had noticed that a locked port could be built but not installed, if
the user answers "yes" to the question whether the lock should be
ignored.

> An improved portmaster arouses my interest.  Maybe modify the name so it can be added to the ports tree and coexist with the "official" portmaster.

After taking over the current portmaster I noticed that it is ancient
in its structure. It used global variables throughout and forked a lot
if sub-processes to allow for local state.

Since it is extremely cumbersom to work on a linear 4000 line shell
script, I re-wrote it from scratch using bash associative arrays and
used that version since the summer of 2019. That version did already
support clean builds in a chroot jail as an option.

But I noticed that better data structures would allow to substantially
speed-up the scanning phase and decided to port that bash based version
to LUA. It can be found in my GH repo, but I'm currently reworking the
planning phase to allow for multiple pre-decessors of a port (e.g. if
A-v1 is integrated into B-v2 and B-v1 is already installed).

My current (not pushed) development version locks up due to such a case
(deskutils/kdepim-apps-libs has been merged into net/akonadi-contacts
and akonadi-contacts is downlevel, leading two an attempt to get an
exclusive lock on the target package name twice, since the two update
tasks are not currently merged into one).

But other than that it has been usable for simple updates for a long
time already.

> Desired features/options would be to keep going rather than stop when one port fails to build, and the ability to install build dependencies, which may be useful for building other software.

Yes, I mark failed updates as such and cancel planned updates of
dependencies, but unrelated updates go on.

> With synth, I had a difficult time getting everything that was built to install, some packages like bison are needed in building other software.

The bash version supported 4 modes: traditional portmaster, delayed
installation of packages not depended on in the update run, jailed
builds with installation of packages after completion of all builds
and repository mode to just create packages in a jail.

I ran into issues when I tried to optimize the disk space used for
the jail by de-installing no longer required build dependencies
from the jail as soon as possible. I needed better data structures
than offered by bash to efficiently express these dependencies.

That was the point where I decided to migrate the code to LUA.

> How is poudriere in that regard?  I never used poudriere, have been intimidated by not wanting to use zfs or dialog4ports, or such an elaborate setup just to update one or a few ports.

I have to use poudriere to test ports before commit, but I do not
like it. The build jails are quickly becoming stale and have to be
rebuilt (causing all previously built packages to be compiled again)
and I often run into issues where a build dependency fails for reasons
that ar enot obvious (e.g. gmake) and I have found that the easiest
method of recovery is to throw away the whole poudriere jail and to
start over with a fresh one.

> Gentoo Linux with portage has "--with-bdeps=y" which installs build dependencies when desired.

Poudriere builds every port in a clean jail with dependencies
installed from previously built packages. This jail is destroyed
when the new package has been saved and then another fresh one
is created for the next port to build.

This process takes advantage of ZFS features (when building on
ZFS, as suggested) and it uses cloned file systems to quickly
create each jail in a known sane state.

> I found that poudriere uses dialog4ports; I much prefer to save options in a file such as Gentoo Linux does with make.conf and (NetBSD) pkgsrc does with mk.conf .

You always can do this - and poudriere is generally used with
options provied in jail-specific make.conf files.

> I once got a royal mess of circular/jumbled dependencies with dialog4ports; cleaning was a major nuisance, nothing simple like editing /etc/mk.conf or /etc/make.conf .

This should not happen and should be reported to the port
maintainers to look into.

> I would like to be free of dialog4ports; the older dialog was worse and messed up my screen.
Yes, I'm often working remotely on a MacBook and have to find a TERM
description that works well with the options dialog.


I have appended a "port" that builds the LUA version I'm working on
as of 2 months ago. Recent changes to rework the planner to deal
with the above mentioned issue has broken previously working features
and I have not pushed any later commit to keep the head version in Git
usable.

That version has incomplete dependency tracking and only supports
variants of "portmaster -a" (especially not "portmaster <port>" as
it does not reliably collect all dependencies in that case).

This version has the lock-up issue described above, but should be
usable otherwise. (The lock-up can be worked around by deleting the
kdepim-apps-libs package before starting portmaster, but I keep my
test system in this state to easily be able to test the new planner
that shall be able to deal with this situation.)


I'd be interested in system configurations and the times reported by

$ time portmaster.lua --no-confirm -n -a

This takes less than 10 seconds real time on my test system which
has a Ryzen 5 3600 CPU and ZFS with a L2ARC on SSD with 2200 ports
and 600 of them being downlevel. But I do not have performance data
from significantly different systems ... (The above command may
start to fetch missing dist files. I'm interested in the times for
a system that already has all required distfiles available.)


To display the operations that would be performed you can use:

$ portmaster.lua --dry-run -a

Regards, STefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20201227/d72a28c6/attachment.sig>