Ports system quality

[Michal Varga]

On Sun, 2011-08-28 at 21:00 -0700, Doug Barton wrote:
> I think it would be a mistake to believe that we don't have any quality
> control at all. I do think it's reasonable to ask whether what we have
> is adequate, and if not, how can it be improved. That's why I'm
> suggesting the stable ports tree idea as a step in that direction.

Now to be a bit more clearer, I didn't mean it in the sense that anyone
can (or will) happily commit random crap to ports just to be done with
it and go to movies, that wasn't my intention to suggest.

By quality control, I meant first *ensuring* that the new port version
will actually do something meaningful, other than, say, segfault
everything depending on it. And not introducing it to the general
population before that is ensured. See below.


> > This is just one of those situations that won't get magically solved by
> > just another ports tree.
> 
> On the contrary, this is *exactly* the kind of thing that my idea would
> catch. More completely, my idea is something along the lines of:
> 
> 1. Establish a baseline of what works with the existing ports tree via
> -exp run(s).
> 2. Branch the tree
> 3. New commits go to the head of the tree
> 4. "Periodically," we do an -exp run with the stable tree plus selected
> updates from head. If the tree is no worse off than the baseline promote
> the tested updates, update the baseline, lather, rinse, repeat.

And there comes the definition of a *working* port. It was pretty easy
back when most ports consisted of five dependencies and like three of
them were Perl. If it compiled, in 99% cases, it also ran.

This no longer works in the scope of huge desktop environments like
Gnome/KDE (or XFCE and other lesser players). Obviously, some server
configurations suffer from this too - think of Java deploys and similar
monstrosities with hundreds of dependencies.

Testing only for "Does it still build?" won't help much anymore if the
new version silently broke one of the APIs and while Apache still runs
with it fine, no desktop user won't be able to boot his OS anymore (that
is, boot into the desktop, which for number of people *is* the OS).

Or let's just create a hypothetical situation:

- We have this popular graphics library called Foo, which is a
dependency of basically everything - web browsers, image viewers,
editors (GIMP, Inkscape, whatever)...

- Port maintainer decides to introduce a new version, but doesn't
actually use or care for all the features Foo provides, so he tests it
with his favorite image viewer, which he maintains too. So all looks
good, port compiles flawlessly, even his vacation photos show ok, so all
is done, right? (Raise hand everyone who just recognized at least 100 of
the last commits where something instantly broke.)

- Except that slowly over days and weeks after introduction, people now
start asking on the list (or even worse, FreeBSD forums that nobody
reads), why is their web browser missing images since the last week, or
why his GIMP crashes every time he tries to edit his work-related 32-bit
per channel designs (there's an internal joke in that).

- While nobody probably cares much about that guy and his missing
browser images, what would you tell to the GIMP guy? That he should have
waited longer before upgrading the (for him, 30 levels deep) "Foo"
dependency? With furious client breathing down his neck and everything?
Yes, he will be eventually able to track the issue down and downgrade
Foo to some working version, only after losing a full work day. I know,
it's all his fault for not getting a Mac, but I'm still being in the
hypothetical land so this still counts.

Now where I'm trying to get by this:

Either we want to have ports as a "big repository of colorful stuff that
even builds", or we want to have some actual products that people can
use after they build them. And that needs an additional level of quality
control that FreeBSD currently, and horribly, lacks (patches welcome, I
know).

I'm not saying that this can be solved overnight and definitely not by
me, but there's at least one thing that can make the world better in
that regard, if some enterprising soul introduced it, I don't know,
maybe over a coming decade:

That is, forcing maintainers to work together to maintain a whole
product, not everyone randomly updating crucial components just because
they feel like it's a nice day to do it.

- That particular maintainer of Foo graphics library should be forced,
by threats of violence, to consult his upgrade with all consumers of his
port (that is, other maintainers of ports that use it as a dependency). 

- Maintainers of those ports should be forced, dunno, possibly by
threats of violence, to actually test that proposed library upgrade with
their ports and approve or block the upgrade (or somehow work together
on a solution, stuff like that).

- AND there should be an enforced infrastructure to make all that
somewhat easy for them, with -exp runs being a part of that process (and
a non-optional part to that).

- And only after all involved parties finish all testing and approve the
dependency upgrade, commit the port for Joe Public to finally start
using it.

Without at least this as a corner stone of quality control, ports will
only keep rotting further, the more the applications will keep getting
complex and entangled in hundreds and hundreds of dependencies. The
simple fact that port builds isn't, and wasn't for a long time now, an
indicator that the port really works. Or that the *ports* really work as
a whole.

m.


-- 
Michal Varga,
Stonehenge (Gmail account)