Misuse of PORTREVISION (Re: svn commit: r434379 - head/multimedia/x265)

[Bryan Drewery]

On 2/21/2017 10:19 AM, Tijl Coosemans wrote:
> Poudriere does not rebuild everything that needs to be rebuilt by the way.
> It just always rebuilds everything, full stop.

I have a lot of replies to this thread and picked this one at random.

1.
First of all, this is not true.  Poudriere *does not* rebuild everything
every time.  It only does that with the -c flag.  Poudriere *by default*
will do an incremental build.  It only rebuilds things if they have,
comparing to the previous package, a missing package, a new PORTVERSION,
a new PORTREVISION, new LIB or RUN dependencies, changed OPTIONS, a
changed PKGNAME, or lastly if a dependency rebuilds due to any of the
previous.

This last piece, of always rebuilding if a dependency is rebuilt is
simply *wrong*.  It is a crutch that was added to ensure that packages
were working because people often *forgot* to bump PORTREVISION properly.

Before I go on I should point out that Pkg uses similar logic.  It will
not reinstall a package that Poudriere aggressively rebuilt.  Doing so
*requires a PORTREVISION* bump.

2.
Shared libraries are not the only cause to need to rebuild a package.
Any number of the *build dependencies* may have been changed to produce
a new output that modifies the resulting package somehow.  Be it runtime
or documentation or other metadata files.

What if Doxygen is updated and now has a drastically new output format
(and generates the same exact file output names/paths so plists are not
impacted), should all ports needing it be rebuilt automatically (which
Poudriere would do) and then have the new documentation format?   Pick
one.  If you think yes, then Poudriere is right and Pkg is wrong.  What
if Doxygen just had a buildtime fix for mips, so now we're rebuilding
x86 packages and the Doxygen output is the same.  It clearly is wrong to
rebuild the packages in this case for x86.  So now we're contradicting
our "yes" from earlier.  In fact we do this *very often* on the package
builders, where we are rebuilding a ton of things that have 0%
difference from the previous except some timestamps.


3.
Blaming portmaster/portupgrade is not correct.  Both of those tools use
95% logic from the Ports framework.  In the case of rebuilds for
dependencies, they would only do it if the DEPENDS line specified a
version requirement that changed and a PORTREVISION bump occurred.  I
forget if either implements shared library auto rebuild logic, but
regardless, the *ports framework* does not.

4.
A reminder that not doing a PORTREVISION chase as-needed will make Pkg
users have broken packages until they force reinstall some of them,
unless it happens to be a shared library change which it will recognize.

5.
What Poudriere does is emphatically wrong as it breaks reproducible
packages.  Even if we don't support that now, the idea that
SOMEPACKAGE-1.0 built last week and built today may have Doxygen 1.0
docs and Doxygen Web2.0 docs respectively, is very wrong.  In my example
this is just documentation, but it could very well be that a package
built last week worked but built today does not work, yet they are the
same exact version and portrevision.  Add in the Pkg problem of not
reinstalling them and you get some random users having problems and
others not, while *both* have the updated dependencies because Pkg did
upgrade those, but those dependencies had *build time* differences that
broke packages.

6.
We could use tools like abi-compliance-checker to automate more of this,
but they still don't track other side-effect changes from build
dependencies, like the Doxygen example.

7.
Using "Pkg with Ports as a backend" would be interesting, but it still
doesn't fix the problem that any kind of static INDEX cannot magically
know if things need a rebuild without things like PORTREVISION.

8.
Lastly, I have always blamed Poudriere's aggressiveness on ports
committers not doing their job correctly with chasing PORTREVISIONS.
There likely are many technical fixes to the problem, but currently none
of the tools (portmaster/portupgrade/poudriere/pkg) do 100% of the right
thing.  I left out synth as I am not familiar with its behavior here,
but it too cannot be magical and know if a package *really* needs a
rebuild or not, and still Pkg may ignore that new one.

I found this thread via a PR to make portupgrade support checking shlib
bumps automatically (which may be fine), and asserting that the
consensus in this thread was that PORTREVISION bumps were wrong.  Well,
no, they are required.  A committer may not like them, but they are
still required.

I wrote about all of these problems in a Journal article a few years
ago.  Nothing has really changed since then.

-- 
Regards,
Bryan Drewery

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/svn-ports-all/attachments/20170222/0269ba9b/attachment.sig>