FCP 20190401-ci_policy: CI policy

[Kristof Provost]

On 29 Aug 2019, at 13:40, Konstantin Belousov wrote:
> On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote:
>> It seems I was doing wrong that just changed the content of this FCP
>> to "feedback", but did not send to the right mailing lists.
>>
>> So I would like to make an announcement that the FCP
>> 20190401-ci_policy "CI policy":
>>
>> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md
>>
>> is officially in "feedback" state to hopefully receive more comments
>> and suggestions, then we can move on for the next FCP state.
>
> What problem does the document tries to solve ?  Or rather, do we 
> really
> have the problem that it claims to solve ?
>
There are, somewhat regularly, commits which break functionality, or at 
the very least tests.
The main objective of this policy proposal is to try to improve overall 
code quality by encouraging and empowering all committers to investigate 
and fix test failures.

>> From my experience, normal peer pressure is enough to get things 
>> fixed
> quickly when it is possible to fix them quickly. If there is something
> more non-trivial, esp. in the tests and not the build, I am sure that
> a rule allowing anybody to do blind revert is much more harmful than
> having a test broken.
>
> More, I know that tests are of very low quality, which means that
> brokeness of the tests is not an indicator of anything until root 
> cause
> is identified.
>
I’m not sure I agree with the characterisation that the tests are of 
low quality. My own experience with the pf tests is that they test a 
large section of the network stack and firewall code. They’ve 
identified several very really issues (both pre- and post commit on the 
epoch-isatin of the network stack, for example, as well as a fairly 
important issue with IPv6 reassembly).
It’s certainly true that the pf tests often reveal issues that are not 
in pf but in other code. I wouldn’t agree that this is a sign of low 
quality tests, but instead I consider it a sign that we don’t have 
enough tests for the network stack itself.

> Can we rely on the common sense of developers until there is indeed 
> the
> visible problem ?
>
I don’t want to suggest that people simply don’t care about test 
failures, because that’s clearly not true.

On the other hand, I do think we can do better. There are at least two 
open problem in the network stack that I currently can’t get anyone to 
look at, and where I personally do not have sufficient context (or time) 
to fix them myself. (#239380, #238870).

This proposal isn’t a silver bullet, I don’t think there is such a 
thing, but I do believe that elevating the visibility and importance of 
test failures can help us improve overall quality.

Best regards,
Kristof