jenkins.FreeBSD.org test characterization/analysis tools

[Garrett Cooper]

Hi all,
	Jenkins provides some limited test failure analysis tools (the history graph [1], the test trend point/bar chart [2], some limited support for determining whether or not a testcase has failed by looking at the last X failures [3] are the only items that it seems we have enabled in our Jenkins install), but characterizing non-consecutive test failures is a bit difficult.
	I took the time to write up a Makefile and some python scripts [4] which pull down all of the test results XML files from Jenkins, do some basic analysis, then summarize the test failures. I’ve included an example below (which I’m going to use in filing bugs for testcases that have proven problematic with Jenkins).
	If this seems interesting, I’ll look at formalizing the API, adding in “pass” results, and doing a few other things (it’s doing similar things to what an internal database at $work is doing tracking test results).
Thanks!
-NGie

PS This information only goes back ~2 months today. Unfortunately all test results prior to that date seemed to have gone missing :(.

$ pwd
/Users/ngie/scratch/tools/analyze-jenkins-failures
$ bmake all
Test = bin.expr/expr_test/overflow, Failed builds = 869
Test = bin.pax/legacy_test/main, Failed builds = 943
Test = bin.pkill/pgrep-j_test/main, Failed builds = 1066
Test = lib.libc.gen/sleep_test/kevent, Failed builds = 1054, 1058
Test = lib.libc.sys/poll_test/poll_3way, Failed builds = 948
Test = lib.libc.sys/sigtimedwait_test/sigtimedwait_NULL_timeout, Failed builds = 1040, 1054-1056
Test = libexec.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, Failed builds = 937, 964, 1010-1041
Test = local.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, Failed builds = 990, 1023-1026, 1037
Test = local.atf.test-programs/expect_test/expect_death, Failed builds = 1032
Test = local.kyua.testers/atf_inttest/test__cleanup__crash, Failed builds = 1053
Test = sbin.devd/client_test/stream, Failed builds = 982, 1064
Test = sys.aio/aio_kqueue_test/main, Failed builds = 983-986
Test = sys.aio/aio_test/aio_fifo_test, Failed builds = 987-989
Test = sys.aio/aio_test/aio_file_test, Failed builds = 987-989
Test = sys.aio/aio_test/aio_md_test, Failed builds = 987-989
Test = sys.aio/aio_test/aio_pipe_test, Failed builds = 987-989
Test = sys.aio/aio_test/aio_pty_test, Failed builds = 987-989
Test = sys.aio/aio_test/aio_unix_socketpair_test, Failed builds = 987-989
Test = sys.aio/lio_kqueue_test/main, Failed builds = 983-986
Test = sys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger, Failed builds = 1061-1071
Test = sys.mqueue/mqueue_test/mqtest1, Failed builds = 983-989
Test = sys.mqueue/mqueue_test/mqtest2, Failed builds = 983-989
Test = sys.mqueue/mqueue_test/mqtest3, Failed builds = 983-989
Test = sys.mqueue/mqueue_test/mqtest4, Failed builds = 983-989
Test = sys.mqueue/mqueue_test/mqtest5, Failed builds = 983-989
Test = usr.bin.grep/grep_test/recurse, Failed builds = 943
Test = usr.sbin.pw/pw_userdel/rmuser_seperate_group, Failed builds = 1045

1. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/history
2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/
3. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/junit/sys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger/ , look for “Failing for the last 11 builds”.
4. https://github.com/yaneurabeya/scratch/tree/master/tools/analyze-jenkins-failures
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-testing/attachments/20150530/6ff76bb5/attachment.sig>