Ports system quality

[Michal Varga]

On Sun, 2011-08-28 at 14:43 -0700, Doug Barton wrote:

> If you're talking about the recent ruby update, an enormous amount of
> work went into that prior to the trigger being pulled in an effort to
> make it as smooth as possible. It's unfortunate that in spite of that
> effort there were still some "issues" that were only discovered after
> users rushed to perform the upgrade. In this case backing out the change
> was the responsible course of action.

I didn't want to mention the Ruby update specifically, because then the
debate would turn into this or that specific issue, and honestly, maybe
a year ago, I'd still think that it's really just about a specific issue
in every single case.

But over a course of time, I'm now inclined to see it more as a full
picture, and that picture is, sadly, "something just constantly being
broken" (yes, in the big picture). No matter how much some individuals
invest into FreeBSD ports, the overall loss of quality control is pretty
visible, and again - no single individual will fix that all. No two of
them, no dozen of them. This is something that has to be fixed on a
higher level, and a nice step in that direction would be a complete
revamp of (currently like, close to none?) quality control procedures,
and then require port maintainers (and commiters) to actually follow
them, every time.

Even if they are volunteers, and everyone contributes on their own time
and resources, sure, but I would, personally, rather have 10000 (or
1000, if there's no other choice) fully working ports again, instead of
those 22000 there all the time untested, broken, or missing features or
introducing regressions every minor update.


> Now, how do we fix this? It has been suggested numerous times that one
> solution to this problem would be a "stable" ports tree. The idea being
> that after changes have had a chance to shake out for a while in the
> head of the ports tree they get merged back to a stable branch. This
> needs to happen, yesterday.

Stable ports tree *alone* as a solution is a waste of time (yes, I know
how ridiculous that statement sounds in this thread, but-). There are
only two possible outcomes of this, one worse than another:

1. Nobody will be using the "testing" tree, because it will be
constantly superbroken (like, much more than ports are now). In theory,
at least for some people, such tree would sound like a great idea to
"get your latest Firefox / Xorg / Gimp / whatever you like", but the
dependency tree you will need to pull will basically overwrite half of
your stable ports tree anyway, so you can as well keep using "testing"
on full time. Which in turn means having everything broken all the time.
Which in turn means, rarely anybody will be using it outside of port
maintainer/commiter circles, which means you won't get any additional
testing anyway.

And / Or:

2. Stable tree will become terribly outdated, as every just a little bit
complicated app comes with 300 dependencies (think of GTK/QT and all the
way down from there). This means - do I want, as a maintainer, promote
this new (and properly tested) Firefox to the stable tree? Well, but I
will need you guys to promote me that latest GTK and right, recent
gstreamer, and, uhmmm, some recent smb stuff and just a few more, and
oh, they won't work without their latest dependencies too so in turn, we
either end up with a ~100 port promote to the stable tree, or we will
wait a few months until all the needed components get there on their
own. So -stable tree will rot and nobody will want to use it because
"man, it has like already three releases old Pidgin, what good is that
for?" (which in fact is a security threat by itself, so probably not the
very best example, but you get the basic idea).

What is needed much more than two different port trees, is a cooldown
period combined with *required* testing by port maintainers (and by port
commiters), with some actual consequences if they fail to do so. Now
this again sounds ridiculous (to some), shooting volunteers against the
wall for breaking rules, so that alone is why this will never happen.

But the fact is, that in the current state of ports, we advertise
something that doesn't really work as a whole. As the current rules go
now, port gets a go as long as it compiles, and that's why we have now
those beautiful 22000 ports to advertise. Except that there is nothing
that requires a port maintainer to know how their port is being used:

Again, I'm not going to point fingers, but there was this little
dependency port once that after one upgrade totally broke about every
single GTK application. How so? Did the maintainer even briefly test it
before submitting the new version? As it eventually turned out in a
private conversation, he actually tested it pretty thoroughly, with one
exception... Port maintainer was a server user and never even ran
FreeBSD as a desktop system. And there is this minor issue, that his
port is also one of the dependencies that gets pulled into every single
GTK application, right? So during his testing, nothing wrong showed up
with the new version, as with the limited (non-GTK) scope that he was
ever using (and testing) his port for, everything was fine. On the other
hand, for desktop users, the new port broke practically everything. Yay.

So how would one prevent situations like these without *forcing*
volunteers to do job out of the scope they volunteered for? What is the
solution for that? Requiring them to test for something they don't even
use, or even know? What's the better option then? Having the guy drop
maintainership if he is not interested in fully maintaining it as a
dependency? Or advertising a working Gnome (KDE, XFCE, whatever) Desktop
on FreeBSD, and having down there someone maintaining a low level
dependency that can completely break the *desktop OS* every single
update, while obviously he is not even testing for it?

This is just one of those situations that won't get magically solved by
just another ports tree. Obviously, one additional "testing" tree will
catch some of the breakages (depending on number of people willing to
deal with unstable ports, and in my humble expectations, that would be
like:

- almost nobody for server configurations (because why)
- almost nobody for desktop configurations, because that would mean
desktop that's not even starting 90% of the time, so why even bother

), but being the visionary I am, I think it won't change as much as it
now shows on a paper. There already are testing repos for various ports
(Gnome, Xorg, VirtualBox, etc.), and barely anyone uses them.

What is needed more are new, much stricter maintainership and committing
policies than there are in effect now, and seriously - who would dare to
even suggest such thing and still live through the mass burning and
exodus following it?


> The other thing that will help between now and then is to manage your
> change windows a little more conservatively. Except for security-related
> updates there is almost always zero reason to upgrade to new versions of
> things immediately after they hit the ports tree. With all due respect
> to those involved, one of the reasons the ruby thing was such a mess was
> that users jumped in and started upgrading stuff without knowing what
> they were doing, or why. Personally as soon as the notice about the
> upcoming change went out I put the knob in make.conf to keep my systems
> at 1.8 to make sure I wasn't affected.

I'd have to strongly disagree with this point of view.

Thing to keep in mind is, that at any point in time, there is always
*something* that had just hit the tree only moments ago. What if I had
those 30 ports to upgrade (or make it 300 if one upgrades once over a
month), all of them being cross/dependencies along the path requiring
some of each other, and now I have to cherry pick them and
blacklist/exclude some based on their arrival time... That's crazy.

Many people haven't even heard about half of those dependencies down the
tree (and why should they), and how many of them even don't know that
Ruby is a programming language (why should they, it's crap anyway) and
that 1.9 is a major upgrade. Why would someone upgrading a web browser,
music player (or more specifically, *bringing all his applications up to
date*) know about some Ruby and what it is and that 1.8 vs 1.9 can also
be a major deal-breaker, when for him, it's just some random dependency
30 levels below?

Thinking about upgrading ports in this way would require every single
port user to become a maintainer of... everything.

What you propose is the opposite of cooldown period - you'd require
every ports user keeping track on every single port installed and know
'how important' in the overall scheme it is and when it's safe to update
it and when you have to schedule a week-long alarm to let it cool down a
bit until it's (probably) safe to use it (as some other people got
probably burned and hopefully reported it).

This is really a bad idea. When ports hit the tree, they should either
be stable and ready to use, or not be there at all.

m.


-- 
Michal Varga,
Stonehenge (Gmail account)