Generating Kyua test result output in JUnit XML format for Jenkins integration?
asomers at freebsd.org
Tue Feb 25 03:06:03 UTC 2014
On Mon, Feb 24, 2014 at 5:58 PM, Craig Rodrigues <rodrigc at freebsd.org> wrote:
> On Mon, Feb 24, 2014 at 3:07 PM, Alan Somers <asomers at freebsd.org> wrote:
>> On Mon, Feb 24, 2014 at 2:11 PM, Craig Rodrigues <rodrigc at freebsd.org>
>> > Hi,
>> > Some Continuous Integration systems like Jenkins
>> > and Atlassian Bamboo can natively parse
>> > a test result format called JUnit XML.
>> > How hard would it be to add some code
>> > to Kyua to optionally be able to generate test result output
>> > in JUnit XML?
>> > Here is some example Junit XML output:
>> > http://nose2.readthedocs.org/en/latest/plugins/junitxml.html
>> > It would be super cool if by BSDCan we could demo:
>> > * start Kyua tests in Jenkins
>> > * have Kyua run the tests and generate the JUnit XML test output
>> > * have the tests results visible in the Jenkins web UI, something like:
>> > http://zbyhoo.files.wordpress.com/2011/05/jenkins-tests.png
>> > How hard would this be to do?
>> > Is there anyone available who is interested in working on this?
>> This is a very awesome idea. Kyua's report-html command is only 286
>> SLOC long, so this probably won't be very hard. Do you have a link to
>> the format specification? I'm afraid that I can't do the work,
>> however. I have extremely limited free time ATM.
> The closest thing I have to a format specification are links to these XSD
> http://windyroad.com.au/dl/Open%20Source/JUnit.xsd (more complicated)
On second thought, I think that Junit-XML output is a good idea in the
short term, but not in the long term. In the long term, displaying
Kyua results in Jenkins is nonideal because:
1) A Kyua run could easily have thousands of test cases, which are
impractical to display in a flat view like Jenkins does. Some sort of
hierarchical view will be necessary.
2) I don't see any "expected failure" results in the links that you
provided. That's an absolute requirement for Kyua. Treating expected
failures like ordinary failures will greatly reduce the utility of
Kyua, by reducing its ability to spot regressions.
3) The Kyua DB format is designed to support viewing a test case's
success history over time, even though the tools don't do it yet.
That's going to be a killer feature when its implemented, but I doubt
that Jenkins has support for that kind of thing. I think that it will
be difficult if not impossible to add it via a Jenkins plugin, but
perhaps I just lack imagination.
In the long term, I think we'll need to write a native webapp for
viewing Kyua results. But in the short term, it will definitely be
easier to get Jenkins integration, so I think that it's worthwhile to
pursue both. It sounds like we'll probably have to read the Jenkins
source to figure out exactly what it thinks JUnit-XML format is.
More information about the freebsd-testing