Automatically running /usr/tests on stable/10 branch under Jenkins

K. Macy kmacy at freebsd.org
Sat Oct 25 20:20:07 UTC 2014


>>> Alan also suggested against integrating the test suite as-is, because as he said, "Remember, don't run these tests on a production system.  They WILL cause panics and deadlocks, and they may cause data loss too.”
>>>
>>> Cheers,
>>> -Garrett
>>
>> Wait, we want to sweep those bugs under the rug?  What exactly is wrong with making a test harness that can very easily reproduce a known problem?  The chances are that anyone will dive into it once the bug is easily reproducible.
>
>         Sweeping bugs under the rug is not what I plan on doing; I’m marking these as expected failures, as opposed to having them continually panic a machine. Once a ZFS dev takes a look at the issue and resolves them, then the ZFS dev can remove the “bail” calls I’m adding to the testcases.
> Cheers,
> -Garrett

Yes, disabling tests that fail leads to an ineffectual test suite. A
test suite that never has any failures is not very useful. However,
there are two factors to take in to account in this context:
a) frequent failures can lead users to stop running a test suite
leading to further regressions
b) long-term repeated failures can desensitize users leading them to
ignore *new* failures facilitating further regressions

Thus it's really a question of what context you're talking about
running the test suite in. For purposes of Jenkins we want full
visibility in to what is passing and what is failing and how long this
has been going on for.

Cheers.

-K


More information about the freebsd-testing mailing list