Automatically running /usr/tests on stable/10 branch under Jenkins

[Craig Rodrigues]

On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein <alfred at freebsd.org>
wrote:

>
> On 10/24/14 9:45 PM, Garrett Cooper wrote:
>
>> I think getting tools/regression/zfs working first would be a better idea
>> (which means that ZFS developers will need to go debug/fix the issue noted
>> in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I'll go
>> ahead and commit my fixes to head from my github fork so it runs.
>>
>
If tools/regression/zfs has Kyuafiles that makes it easier to
run under the kyua tool, that would greatly facilitate running
under Jenkins automation, and would be useful.
I notice some of the fixes you are applying to the regression/zfs
tests is to make certain tests not run on FreeBSD because they
cause known kernel panics such as this:

https://lists.freebsd.org/pipermail/svn-src-all/2014-October/093671.html

I'm not entirely convinced that this is a good way to "fix" a test.
If a test causes a panic, then that's what it does, it it should not be
swept under the rug, as Alfred has pointed out.
Printing out a warning with a pointer to the PR just before running this
type of test is OK, though.



>
>> Alan also suggested against integrating the test suite as-is, because as
>> he said, "Remember, don't run these tests on a production system.  They
>> WILL cause panics and deadlocks, and they may cause data loss too."
>>
>> Cheers,
>> -Garrett
>>
>
> Wait, we want to sweep those bugs under the rug?  What exactly is wrong
> with making a test harness that can very easily reproduce a known problem?
> The chances are that anyone will dive into it once the bug is easily
> reproducible.
>

I agree with Alfred on this.  Even though Alan's test suite
may kernel panic or cause problems, there is still value in running it
and making the results visible on jenkins.freebsd.org.
Running these tests inside a VM which is generated during the build
will allow these types of test to run, but still keep the test machine
usable,
even if the VM gets corrupted while running the tests.

If we have test suites for ZFS, but no one runs them, then no one will
bother to investigate and fix the bugs.

Running the test suites under automation that is visible on
jenkins.freebsd.org is going to force developers to see problems
much sooner than they do now.  Just having the tests in the tree
and hoping that people run them and care to look into the problems is
not enough.

--
Craig