Panicking UUTs and integrating the ZFS test suite into the default run

Julio Merino jmmv at
Mon Oct 27 15:10:40 UTC 2014

On Sat, Oct 25, 2014 at 8:59 PM, Garrett Cooper <yaneurabeya at> wrote:
> (moving to -testing and renaming because this is turning into a bikeshed discussion)
> On Oct 25, 2014, at 9:49, Craig Rodrigues <rodrigc at> wrote:
>> On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein <alfred at> wrote:
>   6
>   7 [ "${os}" = "FreeBSD" ] && die "panics FreeBSD; see bug # 194589"
>   8
> 1. The issue is documented, so developers and testers know where to get the bug information
> 2. The test is marked as a failure.
> Both of these items encourage testers and other developers to go prod developers with knowledge and cycles to go fix the bugs so things eventually turn green.

That's bad as well. Marking tests as broken trains developers to
ignore failures and to be more lax in tolerating breakage, and thus
makes the signal from the test suite useless. Developers must be able
to trust the results of the test suite, and to understand that a flip
from green to red is really bad. Thinking "Hey, the build has been red
like, forever, so one more broken test from me is irrelevant!" is
quite common.

These tests must be kept green if they are for known problems. A way
to do so is with "expected failures", which I think you can denote in
TAP with the "TODO" keyword and with expect_fail in ATF.

Regarding the panics: making the box panic every time the test runs is
probably not a great idea... but this is a shortcoming caused by our
inability to properly test the kernel in an isolated environment. (In
NetBSD, we'd use rump for this and thus the machine would never

More information about the freebsd-testing mailing list