portmaster new development

Mon Dec 28 16:29:57 UTC 2020

On 12/27/20 4:00 AM, LuMiWa via freebsd-ports wrote:
> The subject is 'portmaster new development' but again start pushing
> poudriere to FreeBSD users. I do not use zfs file system and I do not
> use poudriere and I do not want to use on my computer for building some
> ports and then spending hours and hours with poudriere with not enough
> machine. For me is portmaster perfect as is now.

  More tools and improvements to tools can be a good thing. There are
some things that portmaster did better and others that portupgrade did
better. Can't remember which for each task but I think I used portmaster
for configure dialogs and fetching as it would perform those steps
before builds started. I found myself using portmaster to create a list
of ports I had installed (portmaster --list-origins >
~/installed-port-list) to then feed into poudriere as a list to build
though pkg prime-origins completes in a small fraction of a second
instead of more than 5 seconds.
  I've dealt with a number of ports that fail to build because of a
conflict (usually not noted). May have been avahi or a different port
sounding similar in my head but one would regularly fail to upgrade for
me if it was currently installed because the build found and used tools
from the old version during the build of the new even after it spent
time building the new tools in the work directory. People probably think
those issues are rare in the ports tree but back around 2009-2011 I
upgraded a x86 to x64 by upgrading in place. Running portupgrade lead to
many failures due to ports using locally installed 32bit tools and
libraries even though in theory everything should have been
rebuilt/reinstalled in an orderly fashion where it would be replaced
before it gets used. I've thought about repeating that to document
findings but unless other developers want to look into the issues then
there is no point as I have not been able to fix those type of build
issues when I've tried. Deleting any installed package before its own
port rebuild would work around many of those issues I saw but you really
need to uninstall all nondependent ports to guarantee an individual port
build succeeds. If uninstalling a port does not remove everything (bad
pkg-plist) it created then there is still a chance of a contaminated
build. Some builds will break if you have nondependencies installed and
other times it may just modify what you get from building it as the
program detects its surroundings and makes findings the port maintainer
didn't account for when the build autodetects surroundings with things
like autotools which didn't have the maintainer override or add a
dependency for every possible optional manual detection. Remove port(s),
rebuild, and reinstall once built is higher downtime than a pkg upgrade
of already built ports. That is made worse when portmaster aborts on an
unrelated dependency tree branch causing a complete build halt
mid-upgrade. Portupgrade's "continue building all others on fail" leaves
less programs broken without interaction but still higher than if it
rolled back the entire dependency tree of a failed port.
  Building in a jail gets around the issue of needing a separate build
machine, removing installed packages, and having program downtime while
waiting for the chain of it+dependencies to all be rebuilt (incomplete
build of program+dependencies leads to broken programs if I cannot
resolve). That time would be extended when I would have days worth of
building to do and portmaster would stop building within hours or
minutes of starting and I won't see results for hours later when I get
home from work or wake up. I completed very little with portmaster
before going back to portupgrade back in the day because abort on
failure rarely errored out on the first or last of a dependency chain
and stopping in the middle without rolling back the entire chain would
mean the update attempt generally broke things other than the one failed
port. Installing from packages after all building is done offers the
shortest downtime of programs on the system.
  A poudriere build stays in its own area and keeps it clean as it goes
so those problems go away. Other solutions including a
portmaster/portupgrade building in a jail should be able to do so too.
Poudriere can be configured to not use excessive resources by
controlling number of different ports to build at once, how many threads
to use per port build, and how much to use memory disks vs real disks
for workdirs, ccache, etc. Poudriere will take longer due to always
starting with a clean environment, being more aggressive about deciding
when a port needs a rebuild, and time spent building/updating the base
system to be used inside the jail. Other jailed build systems will
likely perform faster due to less cleanup per package. Poudriere
defaults performing a full build for my machine while I use my machine
is a bad experience but had controls to adjust settings to all be more
sane. Poudriere, with more tame settings than default, leads to a bit
longer build time for much less downtime on my desktop with CPU from
2012(?) which does both the building and normal desktop use. It does all
this on spinning rust as I haven't taken up the task of various
filesystem changes I want to make as I upgrade to SSDs even though I've
been talking customers into (and forced family into) SSD upgrades for years.
  Poudriere still offers the fine grained + general changes to
customize/break builds. Last changes I made were trying to enable
options for every port checking them with OPENMP breaking imagemagick
builds and FFMPEG causing problems. I took out unsetting NLS; can't
remember why but probably sorting out problems too; reason for using it
was I only speak English and additional dependencies for unused language
capability was just more bloat likely to slow building more and lead to
more rebuilds when the seemingly unneeded dependency gets an update. I'm
playing with turning on "DTRACE THREADS OPTIMIZED_FLAGS OPTIMIZED_CFLAGS
OPENCL ASM MMX SSE SSE2 SSE3 SSE41 SSE42 AVX" though wouldn't be
surprised to see it is why some videos don't play properly in VLC now,
or could be options I need to review but haven't checked. The more you
change builds from defaults, the more likely you are to find issues
first and not all show up in mailing lists presuming you even notice its
a custom issue as it may just be less stable software instead of a build
failing with an error message to pass to google.
  The following was a command I built up over time to help download
distfiles before running portupgrade. Exclusions were to keep old
versions, keep things not in the ports tree (deleted ports, ports I
worked on but never reached a point of submission, etc.), and ignore
things otherwise causing errors. If doing the same thing, you can build
this command one -x at a time but it gets slower waiting for each
iteration to reach an error as you have to start at the beginning on
every abort which by the end was >10 minutes to just process through the
list of ports to the point of the next failure so takes many hours to
create this command. End result is portmaster was nice to see but a big
waste of time for me though didn't stop me from trying and I do look
forward to upgrades of it even if I try to not use such tools now.

portmaster -af -x cinelerra -x replicatorg-0025 -x mp3encode-1.10_2 -x
electricsheep-2.6.8_7,1 -x ltmdm-1.4_19 -x uppc-kmod-0.8_3 -x
libcapplet-1.4.0.5_11 -x autopsy-2.08 -x festlex-ogi-2.2_1 -x
mjpegtools-uvfilters-1.1_4 -x p5-Getopt-Mixed-1.10 -x wine-1.1.36,1 -x
zsnes-1.51_4,1 -x sphinx-2.0.4_2 -x wavplay-1.4_2 -x gogo-2.39b_1 -x
svgalib -x win32-codecs-3.1.0.r1,1 -x inventor-2.1.5.p10_5 -x
linux-f10-xorg-libs-7.4_1 -x xmlto-0.0.23 -x
system-config-printer-1.0.16 -x gnome-user-docs-2.28.2 -x
gtkhtml-1.1.10_11 -x ocaml-lablgtk2-2.14.0 -x mldonkey-3.0.1_1 -x
portdowngrade-0.6_1 -x strace-4.5.18_1 -x teamspeak_server-2.0.24.01 -x
zgv-5.9_2 -x bonic-einsteinathome-4.18_2,2 -x cpuburn-1.4 -x
bladeenc-0.94.2 -x emerald-themes-0.5.2 -x libcapsinetwork-0.3.0 -x
gcc-4.4.7.20110628 -x monopd-0.9.3 -x koffice-1.6.3_17,2 -x
avr-gcc-4.5.1 -x libmusicbrainz-2.1.5 -x kdelibs-3.5.10_8 -x
libtunepimp-0.5.3_9,1 -x p5-Version-Requirements-0.101021 -x
freepops-0.2.9_2 -x gpgme-0.3.16_8 -x p5-HTTP-Message-6.02 -x
gcc-4.4.7.20120117 -x kde-icons-noia-warm-0.95 -x gcc-4.4.7.20120306 -x
cgal-3.6.1 -x k3b-1.0.5_13 -x kdegraphics-kuickshow-3.5.10_3 -x
mjpegtools-2.0.0_2 -x fnlib-0.5_9 -x libmapi-0.9_2 -x optipng-0.6.5 -x
fuser-1142334561_3 -x portmanager-0.4.1_9 -x freetype-1.3.1_4 -x
libutempter-1.1.5_1 -x gcc-4.4.7,1 -x avogadro-1.0.3_1 -x
evas-engine-xrender-1.0.1 -x libxml-1.8.17_5 -x ja-vflib-2.25.6_12 -x
jdk-1.6.0.3p4_29 -x kde4-freebsd-carddeck-1.0 -x
linux-realplayer-10.0.9.809.20070726_3 -x py27-clutter-1.0.2_3 -x
xmountains-2.9 -x gtkglarea-1.2.3_3 -x kphotoalbum-3.1.1_10 -x
teamspeak_client-2.0.32_2 -x xloadimage-4.1.17_1 -x kde-3.5.10_8 -x
kdeaccessibility-3.5.10_5 -x kdeaddons-3.5.10_5 -x
kdeaddons-atlantikdesigner-3.5.10_4 -x kdeaddon
s-kaddressbook-plugins-3.5.10_6 -x kdeaddons-kate-plugins-3.5.10_6 -x
kdeaddons-kfile-plugins-3.5.10_6 -x kdeaddons-kicker-applets-3.5.10_6 -x
kdeaddons-knewsticker-scripts-3.5.10_4 -x
kdeaddons-konq-plugins-3.5.10_6 -x kdeaddons-ksig-3.5.10_4 -x
kdeaddons-noatun-plugins-3.5.10_4 -x
kdeaddons-renamedlg-plugins-3.5.10_4 -x kdeadmin-3.5.10_4 -x
kdeartwork-3.5.10_6 -x kdeartwork-xscreensaver-kde-3.5.10_7 -x
kdebase-3.5.10_12 -x kdebase-kompmgr-3.5.10_4 -x kdeedu-3.5.10_5 -x
kdegames-3.5.10_4 -x kdegraphics-3.5.10_11 -x
kdegraphics-kamera-3.5.10_4 -x kdegraphics-kooka-3.5.10_4 -x
kdegraphics-kuickshow-3.5.10_4 -x kdelibs-3.5.10_11 -x
kdemultimedia-mpeglib_artsplug-3.5.10_6 -x
kdemultimedia-xine_artsplugin-3.5.10_8 -x kdenetwork-3.5.10_4 -x
kdenetwork-kopete-0.12.8_7 -x kdepim-3.5.10_12 -x kdesdk-3.5.10_8 -x
kdetoys-3.5.10_4 -x kdeutils-3.5.10_10 -x kdewebdev-3.5.10_9,2 -x
dbus-qt3-0.70_6 -x qt-copy-3.3.8_14 -x py27-qt-3.18.2_2,2 -x
akode-plugins-mpc-2.0.2,1 -x akode-plugins-oss-2.0.2,1 -x
arts-1.5.10_8,1 -x artswrapper-1.5.3 -x kde-xdg-env-1.0_3,1 -x
kdehier-1.0_11 -x akode-2.0.2_2,1 -x akode-plugins-ffmpeg-2.0.2_4,1 -x
akode-plugins-jack-2.0.2,1 -x akode-plugins-mpeg-2.0.2,1 -x
akode-plugins-resampler-2.0.2,1 -x akode-plugins-xiph-2.0.2_4,1 -x
k3b-1.0.5_16 -x kde-workspace-4.8.4 -x kdemultimedia-3.5.10_7 -x
libkdcraw-0.1.9_5 -x libkexiv2-0.1.9_8 -x libkipi-0.1.6_6 -x
qca-tls-1.0_8 -x qscintilla-1.7.1_8 -x digikam-0.9.6_4 -x
digikam-doc-0.9.5_4 -x dolphin-0.8.2_6 -x filelight-1.0_7 -x
kmplayer-0.10.0c_8,2 -x imgSeek-0.8.6_15 -x potracegui-1.3.4_7 -x
kmldonkey-0.11_4 -x klibido-0.2.5_9 -x krename-3.0.14_6 -x
kipi-plugins-0.1.7_11,1 -x kdirstat-2.4.4_7 -x qcad-2.0.5.0.1_8 -x
krusader-1.90.0_6 -x libkexif-0.2.5_10 -x qsa-1.1.5_1 -x
linux-realplayer-10.0.9.809.20070726_3