(In)Stability of the Quarterly Branch

[Torsten Zuehlsdorff]

On 16.12.2016 08:24, David Demelier wrote:
> 2016-12-15 17:25 GMT+01:00 Matthew Seaman <matthew at freebsd.org>:
>> On 2016/12/15 16:01, Olivier Duchateau wrote:
>>>> The problem is that there are no tests in FreeBSD ports. All source
>>>> based systems I've tested: pkgsrc, FreeBSD ports, OpenBSD, Gentoo;
>>>> FreeBSD is the one that have the most instability. Not to mention
>>>> committers that commit without testing the port, just look at
>>>> www/redmine to get your point of view on that issue.
>>
>>> Are your serious when you said, there're no tests on FreeBSD ports. I
>>> can tell you Xfce ports are tested with FreeBSD i386 9.3 and amd64
>>> 11.0 machines (on real hardware, no virtualization), and on poudriere
>>> with Gtk+ 3.20 (port version is not not in ports tree, it's defaut
>>> toolkits for the next stable release 4.14).
>>>
>>> For the LXQt desktop is the same thing (tested with official ports
>>> tree Qt5 and which one in plasma5 branch (on KDE repository).
>>>
>>> I'm also working on the Pantheon desktop (desktop environment of
>>> Elementary OS, I use Vala 0.30.2 and Vala 0.34.4, in order to test
>>> stability of applications.
>>>
>>> I use also OpenBSD macppc, it's piece of shit. WebKit browers are
>>> broken, Xfce components crash often, stable branch is outdated, fix
>>> are not propagated in stable branch. Personally I prefer the FreeBSD
>>> scheme, because I'm sure it's quite stable.
>>
>> Most port committers will run compile tests any time they update a port:
>> the better ones will test compilation on all supported FreeBSD versions
>> and all hardware architectures they have access to (ie. generally i386
>> and amd64).
>
> I'm not talking about being sure that the port builds, but that the
> software works. This is a next step that is too often forgotten. For
> example I remember several years ago having a problem with
> audio/mumble. The port was building fine, the window opened fine but
> it was impossible to speak because there was a problem regarding the
> CELT libraries IIRC.

I really support this, but from a committer perspective its quite 
impossible. Often enough we just have no idea how the port needs to work 
- so we must trust the maintainer. But too often there is none.

>> Additionally the package build cluster will rebuild any modified ports
>> within a few days for all of the OS versions and architectures the
>> project tries to provide ports for: that's yet another level of
>> validating the coding of the port itself.
>>
>> However, I believe the OP's point is that *we do not routinely run the
>> software's own built-in regression tests for the packages we succeed in
>> building*.  This is something that is slowly coming.  For instance, you
>> can run 'make test' for many python, ruby or perl packages and see those
>> tests being run.  TEST_DEPENDS is pretty much standardized as the way to
>> install dependencies required for testing nowadays.
>>
>
> Yes, I fully understand the requirements of such tests. I just would
> like that maintainers test the port by building it and *by running*
> them. This is time consuming for sure, but if the maintainers have no
> time, then just keep an old but fully working version :-)

There are two problems with this:
1) Many ports have no maintainer
2) Even more ports have no tests at all

I'm a big fan for testing. I test every software i wrote and in every 
firm i worked i had the lowest bug count. But if you try to teach other 
people to write tests too you hit a hard wall...

So we need to address multiple problems. I'm currently working on an 
project to improve the management of the ports-tree. Hopefully i can 
free some time for all so we can test the runtime much better.

Greetings,
Torsten