Generating Kyua test result output in JUnit XML format for Jenkins integration?

R. Tyler Croy tyler at
Tue Feb 25 03:35:37 UTC 2014

snipped some context, reply still inline

On Mon, 24 Feb 2014, Alan Somers wrote:

> On Mon, Feb 24, 2014 at 5:58 PM, Craig Rodrigues <rodrigc at> wrote:
> >> This is a very awesome idea.  Kyua's report-html command is only 286
> >> SLOC long, so this probably won't be very hard.  Do you have a link to
> >> the format specification?  I'm afraid that I can't do the work,
> >> however.  I have extremely limited free time ATM.
> >
> >
> > The closest thing I have to a format specification are links to these XSD
> > files:
> >
> >
> > (simple)
> >  (more complicated)
> On second thought, I think that Junit-XML output is a good idea in the
> short term, but not in the long term.  In the long term, displaying
> Kyua results in Jenkins is nonideal because:
> 1) A Kyua run could easily have thousands of test cases, which are
> impractical to display in a flat view like Jenkins does.  Some sort of
> hierarchical view will be necessary.

Jenkins already solves for this, if the Kyua generated JUnit-XML uses
dot-delimited ( "class names" then Jenkins will organize things into a
tree-like hierarchy.

Here's an example of the display with Jenkins itself and its 4000+ test cases:

> 2) I don't see any "expected failure" results in the links that you
> provided.  That's an absolute requirement for Kyua.  Treating expected
> failures like ordinary failures will greatly reduce the utility of
> Kyua, by reducing its ability to spot regressions.

JUnit allows for the concept of "skipped" tests, which Jenkins' parsing code
respects. IIRC this is just a "skipped=true" on a <testcase>, I believe that
would help solve for this case.

> 3) The Kyua DB format is designed to support viewing a test case's
> success history over time, even though the tools don't do it yet.
> That's going to be a killer feature when its implemented, but I doubt
> that Jenkins has support for that kind of thing.  I think that it will
> be difficult if not impossible to add it via a Jenkins plugin, but
> perhaps I just lack imagination.

Boy howdy does Jenkins definitely support this! Unfortunately it's somewhat
limited to a Job itself, but you can get pretty good test history down to a
test-case level;

> In the long term, I think we'll need to write a native webapp for
> viewing Kyua results.  But in the short term, it will definitely be
> easier to get Jenkins integration, so I think that it's worthwhile to
> pursue both.  It sounds like we'll probably have to read the Jenkins
> source to figure out exactly what it thinks JUnit-XML format is.

Depending on your level of fear/loathing for Java, I actually disagree that
Kyua should spend time building a custom web app for viewing results. The
Jenkins project supports a *very* rich set of plugin APIs and extension points
to where writing a custom Kyua plugin would be relatively easy.

There's plenty of non-Jenkins examples of generating JUnit XML for what it's
worth, in the Ruby community, the ci_reporter gem generates XML, Cucumber does
natively as well, Python's nosetest-xml package has plenty of examples as well.

I took a look at the C++ in Kyua and I'm afraid I'm not capable enough to write
the reporter, but if somebody wants to ping me on IRC (rtyler) I can probably
help them a bit if they have trouble with the format.

- R. Tyler Croy

     Code: <>
  Chatter: <>

  % gpg --keyserver --recv-key 3F51E16F
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <>

More information about the freebsd-testing mailing list