FCP 20190401-ci_policy: CI policy

[Ed Maste]

On Fri, 30 Aug 2019 at 02:56, Konstantin Belousov <kostikbel at gmail.com> wrote:
>
> When I was (forced to) look into test failures, it was 50 vs. 50 %
> of test bugs vs. some legitimately catched issues.

Certainly if 50% of reported failures are actually test problems
that's much too high. But independent of that, this still suggests the
tests were responsible for reporting a good number of issues in
advance of developers or end users.

> > - The test is difficult to maintain
> This is too.  My main complain is that to debug a test case, I must strip
> all atf* to be able to examine it under a debugger.

Yes, this is my biggest complaint about our current test setup. But
this impacts the tests' friendliness to developers, but not their
efficacy in reporting regressions.