svn commit: r277876 - in projects/zfsd/head/tests/sys/cddl/zfs/tests: cli_root/zpool_upgrade zil

[NGie Cooper]

On Thu, Jan 29, 2015 at 9:18 AM, Will Andrews <will at freebsd.org> wrote:
> On Thu, Jan 29, 2015 at 08:58:57AM -0800, Garrett Cooper wrote:
>>     Please use atf_skip instead of atf_fail so the issues don't count
>> as failures. If you have bug numbers for the issues that would be nice
>> too in the error messages.
>
> To me, skipping tests is only done when the environment is not setup
> correctly for the test.
>
> Panics are a special case in that they cause an irreversible abort of the
> test run itself, and they *are* failures.
>
> We've adopted a practice of marking such tests as 'expected failure' and
> then preemptively failing them.  Does that work for you?

I agree in concept. The only way to really do this in ATF is a two-step:

    atf_expect_fail "This test panics my box"
    atf_fail "Panic!"

This will print out a message like "expected failure: This test panics
my box: Panic!" though in the results, which is incredibly
confusing... That's why Julio, NetBSD, me, and others prefer:

    atf_skip "This test panics my box"

A "filtering" mechanism for dealing with broken/failing tests on
FreeBSD hasn't been developed (and it kinds of invites maintenance
pain as it numbs people to issues in tests). So while the atf_fail is
known and expected for SpectraLogic, it's not for the Jenkins folks,
Isilon, Microsoft, Netflix, Panzura, etc.

> Bugs will come later if I don't have fixes I can commit.

Bugging the symptoms sooner rather than later is better so others know
what commands to avoid entering in (or situations to avoid in general)
when dealing with code. Please, please bug them so others don't
accidentally trip themselves up on the issues you've seen.

PS Thank you again for the work that you, Alan, and the other folks at
SpectraLogic have done! It'll be nice to add some intelligence to
FreeBSD/ZFS with zfsd :).