FCP 20190401-ci_policy: CI policy

[Kristof Provost]

On 29 Aug 2019, at 17:01, Marcelo Araujo wrote:
> Em qui, 29 de ago de 2019 às 22:54, Kristof Provost <kp at freebsd.org>
> escreveu:
>
>> On 29 Aug 2019, at 16:42, Ian Lepore wrote:
>>> (And I don't think breaking a test counts as
>>> breaking the build.)
>>>
>> I fundamentally disagree on this point. A test failure is, just like 
>> a
>> compiler warning, a precious gift that should not be ignored.
>> The more distance (both in terms of time, and in terms of the people
>> involved) there is between a bug being introduced and it being 
>> detected
>> the harder it is to fix it. Test accelerate detection of bugs. If we 
>> do
>> not take test failures seriously (i.e. as an indication something is
>> wrong and should be fixed) the tests will inevitable become useless 
>> in
>> one of two ways: we’ll either disable failing tests (which is what 
>> we
>> tend to do now) reducing test coverage or we’ll have a test suite 
>> with
>> many failures in it, which makes it useless as well. (As with 
>> compiler
>> warnings, the best way to keep them under control is to consider them 
>> to
>> be fatal errors.)
>>
>
> Could you elaborate where is the "fundamentally" you disagree? Where 
> is the
> fundament? You guys are introducing something new, yes everybody knows
> about test, it is year 2019, but nobody can come with new rules tha in
> hours we gonna revert if you "dare to don't fix it". Sorry, this is 
> not how
> people test software and fix it.
>
I do think that breaking a test breaks the build. Something used to work 
and now it doesn’t. That’s breakage, even if it’s not as total as 
it not compiling any more.

>> In either scenario we end up reducing test coverage, which means 
>> we’re
>> going to push more bugs towards users.
>>
>>> I totally agree.  This is an overly-bureaucratic solution in search 
>>> of
>>> a problem.
>>>
>>> If this needs to be addressed at all (and I'm not sure it does), 
>>> then
>>> another sentence or two in bullet item 10 in section 18.1 [*] of the
>>> committer's guide should be enough.  And even then it needn't be
>>> overly-formal and should just mention that if a commit does break 
>>> the
>>> build the committer is expected to be responsive to that problem and
>>> the commit might get reverted if they're unresponsive.  I don't 
>>> think
>>> we need schedules.
>>>
>> I do feel that’s a better argument. We’ve always had a policy of
>> reverting on request (AIUI), so this is more or less trying to be a
>> strong restatement of that, more than a fundamental shift in policy.
>>
>
> We don't have a policy to revert commit, actually revert commit is
> something bad, it is kind of punishment, I have been there, nobody 
> wants to
> be there. Stop to push this non-sense argument.
>
That’s how I’ve interpreted ’11. Developer Relations’ in
in 
https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/article.html#conventions 
  (Specifically, “ If a commit does results in controversy erupting, 
it may be advisable to consider backing the change out again until the 
matter is settled.”)

I understand that it’s not fun to see changes reverted, and it’s 
certainly not the intention to make that the preferred solution. 
That’s why the FCP discusses adds waiting periods and discussion with 
the committer.

Best regards,
Kristof