FCP 20190401-ci_policy: CI policy
Konstantin Belousov
kostikbel at gmail.com
Sat Aug 31 08:55:16 UTC 2019
On Fri, Aug 30, 2019 at 09:54:41PM -0400, Ed Maste wrote:
> On Fri, 30 Aug 2019 at 02:56, Konstantin Belousov <kostikbel at gmail.com> wrote:
> >
> > When I was (forced to) look into test failures, it was 50 vs. 50 %
> > of test bugs vs. some legitimately catched issues.
>
> Certainly if 50% of reported failures are actually test problems
> that's much too high.
50% of what I looked at. The sample size was around 10.
> But independent of that, this still suggests the
> tests were responsible for reporting a good number of issues in
> advance of developers or end users.
>
> > > - The test is difficult to maintain
> > This is too. My main complain is that to debug a test case, I must strip
> > all atf* to be able to examine it under a debugger.
>
> Yes, this is my biggest complaint about our current test setup. But
> this impacts the tests' friendliness to developers, but not their
> efficacy in reporting regressions.
More information about the freebsd-hackers
mailing list