Schedule for releases

[Robert N. M. Watson]

On 22 Dec 2010, at 07:21, Erik Cederstrand wrote:

> Den 21/12/2010 kl. 23.28 skrev Robert Watson:
> 
>> Looking at 7.x, I'm struck by how much it has slowed down.  There's a significant user community, but not a significant developer community. 
> 
> Which pretty much sums up a dilemma in the development of FreeBSD, I think. Developers want users to try out their new shiny stuff, but users don't want to spend time upgrading.
> 
> I think one of many things that would be great to do is to improve the usability and coverage of the regression tests. This would take at least some of the burden off developers who want to MFC their work. We already have the tinderboxes, Coverity and Clang Static Analyzer, but apart from pho's stress tests we don't have any automated runtime testing (as far as I know).

We have an increasing quantity of moderately decent regression tests in our tools/regression tree, but they are only run selectively and moderately rarely. A very nice contribution someone could make would be:

(1) Make it so a recursive regression run is easily accessible -- "make tests ; make runtests" -- in the root or similar.
(2) Set up a "test tinderbox", perhaps using a hosted server at Sentex, and perhaps using virtual machines so that many versions can be run.
(3) Arrange a report website and report e-mails identifying the introduction of regressions in branches.

One known problem with our regression suite is that it includes tests that crash certain versions (and ideally not later versions), so we'd need something flexible enough to skip tests known to fail on particular branches in some usefully configurable way.

All this would be extremely useful -- we can then start pulling in more regression tests written by other communities (other *BSD, etc), and push people to produce more tests when they fix bugs. It would be especially nice to run our security regression tests regularly (testing permissions, ACL semantics, inter-process protections, etc).

Robert