Panicking UUTs and integrating the ZFS test suite into the default run

[Julio Merino]

On Sat, Oct 25, 2014 at 8:59 PM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
> (moving to -testing and renaming because this is turning into a bikeshed discussion)
>
> On Oct 25, 2014, at 9:49, Craig Rodrigues <rodrigc at freebsd.org> wrote:
>
>> On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein <alfred at freebsd.org> wrote:
>
>   6
>   7 [ "${os}" = "FreeBSD" ] && die "panics FreeBSD; see bug # 194589"
>   8
>
> 1. The issue is documented, so developers and testers know where to get the bug information
> 2. The test is marked as a failure.
>
> Both of these items encourage testers and other developers to go prod developers with knowledge and cycles to go fix the bugs so things eventually turn green.

That's bad as well. Marking tests as broken trains developers to
ignore failures and to be more lax in tolerating breakage, and thus
makes the signal from the test suite useless. Developers must be able
to trust the results of the test suite, and to understand that a flip
from green to red is really bad. Thinking "Hey, the build has been red
like, forever, so one more broken test from me is irrelevant!" is
quite common.

These tests must be kept green if they are for known problems. A way
to do so is with "expected failures", which I think you can denote in
TAP with the "TODO" keyword and with expect_fail in ATF.

Regarding the panics: making the box panic every time the test runs is
probably not a great idea... but this is a shortcoming caused by our
inability to properly test the kernel in an isolated environment. (In
NetBSD, we'd use rump for this and thus the machine would never
panic.)