FCP 20190401-ci_policy: CI policy

[Kristof Provost]

On 29 Aug 2019, at 16:42, Ian Lepore wrote:
> (And I don't think breaking a test counts as
> breaking the build.)
>
I fundamentally disagree on this point. A test failure is, just like a 
compiler warning, a precious gift that should not be ignored.
The more distance (both in terms of time, and in terms of the people 
involved) there is between a bug being introduced and it being detected 
the harder it is to fix it. Test accelerate detection of bugs. If we do 
not take test failures seriously (i.e. as an indication something is 
wrong and should be fixed) the tests will inevitable become useless in 
one of two ways: we’ll either disable failing tests (which is what we 
tend to do now) reducing test coverage or we’ll have a test suite with 
many failures in it, which makes it useless as well. (As with compiler 
warnings, the best way to keep them under control is to consider them to 
be fatal errors.)

In either scenario we end up reducing test coverage, which means we’re 
going to push more bugs towards users.

> I totally agree.  This is an overly-bureaucratic solution in search of
> a problem.
>
> If this needs to be addressed at all (and I'm not sure it does), then
> another sentence or two in bullet item 10 in section 18.1 [*] of the
> committer's guide should be enough.  And even then it needn't be
> overly-formal and should just mention that if a commit does break the
> build the committer is expected to be responsive to that problem and
> the commit might get reverted if they're unresponsive.  I don't think
> we need schedules.
>
I do feel that’s a better argument. We’ve always had a policy of 
reverting on request (AIUI), so this is more or less trying to be a 
strong restatement of that, more than a fundamental shift in policy.

Best regards,
Kristof