FCP 20190401-ci_policy: CI policy

[Simon J. Gerraty]

Ed Maste <emaste at freebsd.org> wrote:
> > > - The test is difficult to maintain
> > This is too.  My main complain is that to debug a test case, I must strip
> > all atf* to be able to examine it under a debugger.

IMO this is a bug in the state of ATF in FreeBSD.
The ability to run atf tests on the host in-line with the build is one
of the reasons I picked ATF for Junos - but has been discarded in FreeBSD.
With atf tests runnable on build host in situ they are not hard to debug
there too.

I recently added a test suite for my package system (for Junos) and had
to use the bmake framework, because FreeBSD made it impossible to use
ATF/Kyua

> 
> Yes, this is my biggest complaint about our current test setup. But
> this impacts the tests' friendliness to developers, but not their
> efficacy in reporting regressions.