Automatically running /usr/tests on stable/10 branch under Jenkins

Garrett Cooper yaneurabeya at gmail.com
Sat Oct 25 21:46:48 UTC 2014


On Oct 25, 2014, at 13:20, K. Macy <kmacy at freebsd.org> wrote:

>>>> Alan also suggested against integrating the test suite as-is, because as he said, "Remember, don't run these tests on a production system.  They WILL cause panics and deadlocks, and they may cause data loss too.”
>>>> 
>>>> Cheers,
>>>> -Garrett
>>> 
>>> Wait, we want to sweep those bugs under the rug?  What exactly is wrong with making a test harness that can very easily reproduce a known problem?  The chances are that anyone will dive into it once the bug is easily reproducible.
>> 
>>        Sweeping bugs under the rug is not what I plan on doing; I’m marking these as expected failures, as opposed to having them continually panic a machine. Once a ZFS dev takes a look at the issue and resolves them, then the ZFS dev can remove the “bail” calls I’m adding to the testcases.
> 
> Yes, disabling tests that fail leads to an ineffectual test suite. A
> test suite that never has any failures is not very useful. However,
> there are two factors to take in to account in this context:
> a) frequent failures can lead users to stop running a test suite
> leading to further regressions
> b) long-term repeated failures can desensitize users leading them to
> ignore *new* failures facilitating further regressions
> 
> Thus it's really a question of what context you're talking about
> running the test suite in. For purposes of Jenkins we want full
> visibility in to what is passing and what is failing and how long this
> has been going on for.

(seeing as how my other post isn’t in the -testing archives yet..)
	Panicking a node (what the tests are doing before last night) and exiting with a non-zero exit code (what I’m making them do with the bail outs in tools/regression/zfs) are both considered test failures. The difference being that I can safely run all of the tests on a production or a test machine without having to panic/reboot the box and I get greater coverage in one fell swoop. If a developer wants they can always delete the lines that bail out of the tests to get the desired panic.
Cheers,
-Garrett
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20141025/c81536a1/attachment.sig>


More information about the freebsd-stable mailing list