Ports system quality

[Doug Barton]

On 08/28/2011 19:43, Michal Varga wrote:
> On Sun, 2011-08-28 at 14:43 -0700, Doug Barton wrote:
> 
>> If you're talking about the recent ruby update, an enormous amount of
>> work went into that prior to the trigger being pulled in an effort to
>> make it as smooth as possible. It's unfortunate that in spite of that
>> effort there were still some "issues" that were only discovered after
>> users rushed to perform the upgrade. In this case backing out the change
>> was the responsible course of action.
> 
> I didn't want to mention the Ruby update specifically, because then the
> debate would turn into this or that specific issue, and honestly, maybe
> a year ago, I'd still think that it's really just about a specific issue
> in every single case.
> 
> But over a course of time, I'm now inclined to see it more as a full
> picture, and that picture is, sadly, "something just constantly being
> broken" (yes, in the big picture).

I can certainly understand how you could come to that conclusion, but
like I said, that hasn't been even close to my experience, nor do I
think it's representative of even a significant percentage of our users.

> No matter how much some individuals
> invest into FreeBSD ports, the overall loss of quality control is pretty
> visible, and again - no single individual will fix that all. No two of
> them, no dozen of them. This is something that has to be fixed on a
> higher level, and a nice step in that direction would be a complete
> revamp of (currently like, close to none?) quality control procedures,
> and then require port maintainers (and commiters) to actually follow
> them, every time.

I think it would be a mistake to believe that we don't have any quality
control at all. I do think it's reasonable to ask whether what we have
is adequate, and if not, how can it be improved. That's why I'm
suggesting the stable ports tree idea as a step in that direction.

> Even if they are volunteers, and everyone contributes on their own time
> and resources, sure, but I would, personally, rather have 10000 (or
> 1000, if there's no other choice) fully working ports again, instead of
> those 22000 there all the time untested, broken, or missing features or
> introducing regressions every minor update.

Now you're talking! :)  This is one of the reasons I have been so
supportive of (and made some minor contributions to) the effort to
deprecate stale ports. There is also a lot more work to be done in this
area. That said, it's you and me against all of the (very!) vocal people
who believe that we shouldn't ever deprecate anything. Go figure. :)

>> Now, how do we fix this? It has been suggested numerous times that one
>> solution to this problem would be a "stable" ports tree. The idea being
>> that after changes have had a chance to shake out for a while in the
>> head of the ports tree they get merged back to a stable branch. This
>> needs to happen, yesterday.
> 
> Stable ports tree *alone* as a solution is a waste of time (yes, I know
> how ridiculous that statement sounds in this thread, but-). There are
> only two possible outcomes of this, one worse than another:
> 
> 1. Nobody will be using the "testing" tree, because it will be
> constantly superbroken (like, much more than ports are now).

I'd like to think that we can do a little better than that. :) I also
think that a non-trivial number of users will want to use the "latest
and greatest" so I'm not quite as pessimistic about this as you are.

> And / Or:
> 
> 2. Stable tree will become terribly outdated, as every just a little bit
> complicated app comes with 300 dependencies (think of GTK/QT and all the
> way down from there). This means - do I want, as a maintainer, promote
> this new (and properly tested) Firefox to the stable tree? Well, but I
> will need you guys to promote me that latest GTK and right, recent
> gstreamer, and, uhmmm, some recent smb stuff and just a few more, and
> oh, they won't work without their latest dependencies too so in turn, we
> either end up with a ~100 port promote to the stable tree, or we will
> wait a few months until all the needed components get there on their
> own. So -stable tree will rot and nobody will want to use it because
> "man, it has like already three releases old Pidgin, what good is that
> for?" (which in fact is a security threat by itself, so probably not the
> very best example, but you get the basic idea).

This I think is a very valid concern, and one that will have to be
addressed with vigor. That said, I think that there are a lot of users
who would find value in a ports tree that is overwhelmingly stable, as
opposed to always being the latest and greatest. Also, see below.

> Again, I'm not going to point fingers, but there was this little
> dependency port once that after one upgrade totally broke about every
> single GTK application. How so? Did the maintainer even briefly test it
> before submitting the new version? As it eventually turned out in a
> private conversation, he actually tested it pretty thoroughly, with one
> exception... Port maintainer was a server user and never even ran
> FreeBSD as a desktop system. And there is this minor issue, that his
> port is also one of the dependencies that gets pulled into every single
> GTK application, right? So during his testing, nothing wrong showed up
> with the new version, as with the limited (non-GTK) scope that he was
> ever using (and testing) his port for, everything was fine. On the other
> hand, for desktop users, the new port broke practically everything. Yay.
> 
> So how would one prevent situations like these without *forcing*
> volunteers to do job out of the scope they volunteered for? What is the
> solution for that? Requiring them to test for something they don't even
> use, or even know? What's the better option then? Having the guy drop
> maintainership if he is not interested in fully maintaining it as a
> dependency? Or advertising a working Gnome (KDE, XFCE, whatever) Desktop
> on FreeBSD, and having down there someone maintaining a low level
> dependency that can completely break the *desktop OS* every single
> update, while obviously he is not even testing for it?
> 
> This is just one of those situations that won't get magically solved by
> just another ports tree.

On the contrary, this is *exactly* the kind of thing that my idea would
catch. More completely, my idea is something along the lines of:

1. Establish a baseline of what works with the existing ports tree via
-exp run(s).
2. Branch the tree
3. New commits go to the head of the tree
4. "Periodically," we do an -exp run with the stable tree plus selected
updates from head. If the tree is no worse off than the baseline promote
the tested updates, update the baseline, lather, rinse, repeat.


-- 

	Nothin' ever doesn't change, but nothin' changes much.
			-- OK Go

	Breadth of IT experience, and depth of knowledge in the DNS.
	Yours for the right price.  :)  http://SupersetSolutions.com/