FCP 20190401-ci_policy: CI policy

[Konstantin Belousov]

On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote:
> On 29 Aug 2019, at 13:40, Konstantin Belousov wrote:
> > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote:
> >> It seems I was doing wrong that just changed the content of this FCP
> >> to "feedback", but did not send to the right mailing lists.
> >>
> >> So I would like to make an announcement that the FCP
> >> 20190401-ci_policy "CI policy":
> >>
> >> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md
> >>
> >> is officially in "feedback" state to hopefully receive more comments
> >> and suggestions, then we can move on for the next FCP state.
> >
> > What problem does the document tries to solve ?  Or rather, do we 
> > really
> > have the problem that it claims to solve ?
> >
> There are, somewhat regularly, commits which break functionality, or at 
> the very least tests.
> The main objective of this policy proposal is to try to improve overall 
> code quality by encouraging and empowering all committers to investigate 
> and fix test failures.
But this policy does not encourage, if anything.
It gives a free ticket to revert, discouraging committers.

> 
> >> From my experience, normal peer pressure is enough to get things 
> >> fixed
> > quickly when it is possible to fix them quickly. If there is something
> > more non-trivial, esp. in the tests and not the build, I am sure that
> > a rule allowing anybody to do blind revert is much more harmful than
> > having a test broken.
> >
> > More, I know that tests are of very low quality, which means that
> > brokeness of the tests is not an indicator of anything until root 
> > cause
> > is identified.
> >
> I’m not sure I agree with the characterisation that the tests are of 
> low quality. My own experience with the pf tests is that they test a 
> large section of the network stack and firewall code. They’ve 
> identified several very really issues (both pre- and post commit on the 
> epoch-isatin of the network stack, for example, as well as a fairly 
> important issue with IPv6 reassembly).
This is my experience with the tests for kernel/libc/libthr, most of which
comes from contrib/netbsd-tests, as I understand.

> It’s certainly true that the pf tests often reveal issues that are not 
> in pf but in other code. I wouldn’t agree that this is a sign of low 
> quality tests, but instead I consider it a sign that we don’t have 
> enough tests for the network stack itself.
> 
> > Can we rely on the common sense of developers until there is indeed 
> > the
> > visible problem ?
> >
> I don’t want to suggest that people simply don’t care about test 
> failures, because that’s clearly not true.
> 
> On the other hand, I do think we can do better. There are at least two 
> open problem in the network stack that I currently can’t get anyone to 
> look at, and where I personally do not have sufficient context (or time) 
> to fix them myself. (#239380, #238870).
> 
> This proposal isn’t a silver bullet, I don’t think there is such a 
> thing, but I do believe that elevating the visibility and importance of 
> test failures can help us improve overall quality.
This is not what was proposed.

I fully agree that build failures should be fixed ASAP, and the each
test results change (note the formulation) should be examined.  But
I do not think we have the problem right now of a scale which requires
the free pass to revert with 4h timer.