Generating Kyua test result output in JUnit XML format for Jenkins integration?

[R. Tyler Croy]

snipped some context, reply still inline


On Mon, 24 Feb 2014, Alan Somers wrote:

> On Mon, Feb 24, 2014 at 5:58 PM, Craig Rodrigues <rodrigc at freebsd.org> wrote:
> >> This is a very awesome idea.  Kyua's report-html command is only 286
> >> SLOC long, so this probably won't be very hard.  Do you have a link to
> >> the format specification?  I'm afraid that I can't do the work,
> >> however.  I have extremely limited free time ATM.
> >
> >
> > The closest thing I have to a format specification are links to these XSD
> > files:
> >
> > https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd
> > (simple)
> > http://windyroad.com.au/dl/Open%20Source/JUnit.xsd  (more complicated)
> 
> On second thought, I think that Junit-XML output is a good idea in the
> short term, but not in the long term.  In the long term, displaying
> Kyua results in Jenkins is nonideal because:
> 
> 1) A Kyua run could easily have thousands of test cases, which are
> impractical to display in a flat view like Jenkins does.  Some sort of
> hierarchical view will be necessary.


Jenkins already solves for this, if the Kyua generated JUnit-XML uses
dot-delimited (foo.bar) "class names" then Jenkins will organize things into a
tree-like hierarchy.

Here's an example of the display with Jenkins itself and its 4000+ test cases:
    <https://ci.jenkins-ci.org/job/jenkins_main_trunk/3192/testReport/>



> 2) I don't see any "expected failure" results in the links that you
> provided.  That's an absolute requirement for Kyua.  Treating expected
> failures like ordinary failures will greatly reduce the utility of
> Kyua, by reducing its ability to spot regressions.


JUnit allows for the concept of "skipped" tests, which Jenkins' parsing code
respects. IIRC this is just a "skipped=true" on a <testcase>, I believe that
would help solve for this case.


> 3) The Kyua DB format is designed to support viewing a test case's
> success history over time, even though the tools don't do it yet.
> That's going to be a killer feature when its implemented, but I doubt
> that Jenkins has support for that kind of thing.  I think that it will
> be difficult if not impossible to add it via a Jenkins plugin, but
> perhaps I just lack imagination.


Boy howdy does Jenkins definitely support this! Unfortunately it's somewhat
limited to a Job itself, but you can get pretty good test history down to a
test-case level;
    <https://ci.jenkins-ci.org/job/jenkins_main_trunk/3192/testReport/hudson.tasks.test/TestObjectTest/testSafe/history/>


> In the long term, I think we'll need to write a native webapp for
> viewing Kyua results.  But in the short term, it will definitely be
> easier to get Jenkins integration, so I think that it's worthwhile to
> pursue both.  It sounds like we'll probably have to read the Jenkins
> source to figure out exactly what it thinks JUnit-XML format is.


Depending on your level of fear/loathing for Java, I actually disagree that
Kyua should spend time building a custom web app for viewing results. The
Jenkins project supports a *very* rich set of plugin APIs and extension points
to where writing a custom Kyua plugin would be relatively easy.

There's plenty of non-Jenkins examples of generating JUnit XML for what it's
worth, in the Ruby community, the ci_reporter gem generates XML, Cucumber does
natively as well, Python's nosetest-xml package has plenty of examples as well.


I took a look at the C++ in Kyua and I'm afraid I'm not capable enough to write
the reporter, but if somebody wants to ping me on IRC (rtyler) I can probably
help them a bit if they have trouble with the format.

- R. Tyler Croy

------------------------------------------------------
     Code: <https://github.com/rtyler>
  Chatter: <https://twitter.com/agentdero>

  % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F
------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-testing/attachments/20140224/edce4661/attachment.sig>