FCP 20190401-ci_policy: CI policy

[Konstantin Belousov]

On Fri, Aug 30, 2019 at 09:54:41PM -0400, Ed Maste wrote:
> On Fri, 30 Aug 2019 at 02:56, Konstantin Belousov <kostikbel at gmail.com> wrote:
> >
> > When I was (forced to) look into test failures, it was 50 vs. 50 %
> > of test bugs vs. some legitimately catched issues.
> 
> Certainly if 50% of reported failures are actually test problems
> that's much too high.
50% of what I looked at.  The sample size was around 10.

> But independent of that, this still suggests the
> tests were responsible for reporting a good number of issues in
> advance of developers or end users.
> 
> > > - The test is difficult to maintain
> > This is too.  My main complain is that to debug a test case, I must strip
> > all atf* to be able to examine it under a debugger.
> 
> Yes, this is my biggest complaint about our current test setup. But
> this impacts the tests' friendliness to developers, but not their
> efficacy in reporting regressions.