FCP 20190401-ci_policy: CI policy

[Warner Losh]

On Thu, Aug 29, 2019 at 8:42 AM Ian Lepore <ian at freebsd.org> wrote:

> On Thu, 2019-08-29 at 14:40 +0300, Konstantin Belousov wrote:
> > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote:
> > > It seems I was doing wrong that just changed the content of this FCP
> > > to "feedback", but did not send to the right mailing lists.
> > >
> > > So I would like to make an announcement that the FCP
> > > 20190401-ci_policy "CI policy":
> > >
> > > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md
> > >
> > > is officially in "feedback" state to hopefully receive more comments
> > > and suggestions, then we can move on for the next FCP state.
> >
> > What problem does the document tries to solve ?  Or rather, do we really
> > have the problem that it claims to solve ?
> >
> > From my experience, normal peer pressure is enough to get things fixed
> > quickly when it is possible to fix them quickly. If there is something
> > more non-trivial, esp. in the tests and not the build, I am sure that
> > a rule allowing anybody to do blind revert is much more harmful than
> > having a test broken.
> >
> > More, I know that tests are of very low quality, which means that
> > brokeness of the tests is not an indicator of anything until root cause
> > is identified.
> >
> > Can we rely on the common sense of developers until there is indeed the
> > visible problem ?
> >
>
> I totally agree.  This is an overly-bureaucratic solution in search of
> a problem.
>
> If this needs to be addressed at all (and I'm not sure it does), then
> another sentence or two in bullet item 10 in section 18.1 [*] of the
> committer's guide should be enough.  And even then it needn't be
> overly-formal and should just mention that if a commit does break the
> build the committer is expected to be responsive to that problem and
> the commit might get reverted if they're unresponsive.  I don't think
> we need schedules.  (And I don't think breaking a test counts as
> breaking the build.)
>
> [*]
> https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/rules.html


There's been growing friction around these points in the past several
years. We should document that tree breakages aren't acceptable, committers
are expected to fix things as soon as possible, and that new broken tests
have to be investigated quickly, or the change should be reverted. This
document tries to attach some suggested time frames on different types of
breakage we've seen and set the expectations for people when things go
wrong. Having different categories also allows us to set expectations for
external toolchain breakage that's more lax than in-tree toolchain
breakage, for example. The future will have external toolchain CI and
notifications will likely be turned on when there's breakage.

While some tests are poorly written, we should disable / remove the ones
that are actually bad rather than create a policy that tolerates bad tests
by setting no expectation around that.

Now, one may quibble over the timelines, but that discussion helps
everybody in the community know what the expectations are when committing
and enables people that want to scrimp on testing to do so if they know
they will be around for any unanticipated fallout, but may also encourage
others to test more because they know they might not want to be.

Warner