(In)Stability of the Quarterly Branch

[David Demelier]

2016-12-15 17:25 GMT+01:00 Matthew Seaman <matthew at freebsd.org>:
> On 2016/12/15 16:01, Olivier Duchateau wrote:
>>> The problem is that there are no tests in FreeBSD ports. All source
>>> based systems I've tested: pkgsrc, FreeBSD ports, OpenBSD, Gentoo;
>>> FreeBSD is the one that have the most instability. Not to mention
>>> committers that commit without testing the port, just look at
>>> www/redmine to get your point of view on that issue.
>
>> Are your serious when you said, there're no tests on FreeBSD ports. I
>> can tell you Xfce ports are tested with FreeBSD i386 9.3 and amd64
>> 11.0 machines (on real hardware, no virtualization), and on poudriere
>> with Gtk+ 3.20 (port version is not not in ports tree, it's defaut
>> toolkits for the next stable release 4.14).
>>
>> For the LXQt desktop is the same thing (tested with official ports
>> tree Qt5 and which one in plasma5 branch (on KDE repository).
>>
>> I'm also working on the Pantheon desktop (desktop environment of
>> Elementary OS, I use Vala 0.30.2 and Vala 0.34.4, in order to test
>> stability of applications.
>>
>> I use also OpenBSD macppc, it's piece of shit. WebKit browers are
>> broken, Xfce components crash often, stable branch is outdated, fix
>> are not propagated in stable branch. Personally I prefer the FreeBSD
>> scheme, because I'm sure it's quite stable.
>
> Most port committers will run compile tests any time they update a port:
> the better ones will test compilation on all supported FreeBSD versions
> and all hardware architectures they have access to (ie. generally i386
> and amd64).
>

I'm not talking about being sure that the port builds, but that the
software works. This is a next step that is too often forgotten. For
example I remember several years ago having a problem with
audio/mumble. The port was building fine, the window opened fine but
it was impossible to speak because there was a problem regarding the
CELT libraries IIRC. That a port build is nice, that it works is
better. And it's the same thing for www/redmine, each time I install
it on a fresh machine, `service redmine start` won't start (after
configuring of course) because the Gemfile is broken again. These are
not the only ones. Glade also suffers a bug that makes it almost
unusable.

> Additionally the package build cluster will rebuild any modified ports
> within a few days for all of the OS versions and architectures the
> project tries to provide ports for: that's yet another level of
> validating the coding of the port itself.
>
> However, I believe the OP's point is that *we do not routinely run the
> software's own built-in regression tests for the packages we succeed in
> building*.  This is something that is slowly coming.  For instance, you
> can run 'make test' for many python, ruby or perl packages and see those
> tests being run.  TEST_DEPENDS is pretty much standardized as the way to
> install dependencies required for testing nowadays.
>

Yes, I fully understand the requirements of such tests. I just would
like that maintainers test the port by building it and *by running*
them. This is time consuming for sure, but if the maintainers have no
time, then just keep an old but fully working version :-)

Regards,

-- 
Demelier David