Call for Help: openjdk8 tests under Continuous Integration

Brian Gardner openjdk at getsnappy.com
Fri Nov 28 18:36:53 UTC 2014 

Hi Craig,
I’ve have some scripts setup.  The jtreg produces an xml file for each jtreg test and a browsable html structure.
  The process is made up a two scripts that aren’t very robust at the moment.  The first script starts up poudriere jail with the testport -I openjdk8 options, copies the second script and jtreg into the jail, executes it inside the jail, then shuts down the jail.  The second script just executes jtreg for each subsystem within openjdk that has tests with the appropriate arguments.

I might need to modify the scripts depending on how we integrate them into Jenkins.  Is there somewhere I should commit these too?  What are the next steps to integrating them?


Below is an example jtreg xml file

<?xml version="1.0" encoding="US-ASCII" ?>
<testsuite errors="0" failures="0" tests="1" hostname="localhost" name="vm/verifier/VerifyStackForExceptionHandlers.java" time="0.392" timestamp="2014-11-331T23:55:32+0000">
    <properties>
        <property name="author" value="Keith McGuigan" />
        <property name="keywords" value="bug6547378" />
        <property name="run" value="ASSUMED_ACTION main VerifyStackForExceptionHandlers
" />
        <property name="source" value="VerifyStackForExceptionHandlers.java" />
        <property name="title" value="Verify that methods with max_stack==0 don't have exception handlers" />
        <property name="description" value="file:/wrkdirs/usr/ports/java/openjdk8/work/openjdk/jdk/test/vm/verifier/VerifyStackForExceptionHandlers.java" />
        <property name="elapsed" value="392 0:00:00.392" />
        <property name="end" value="Thu Nov 27 23:55:32 GMT 2014" />
        <property name="environment" value="regtest" />
        <property name="execStatus" value="Passed. Execution successful" />
        <property name="hostname" value="localhost" />
        <property name="javatestOS" value="FreeBSD 10.0-RELEASE (amd64)" />
        <property name="javatestVersion" value="4.4.1" />
        <property name="jtregVersion" value="jtreg 4.1 dev b10" />
        <property name="script" value="com.sun.javatest.regtest.RegressionScript " />
        <property name="sections" value="script_messages build compile main" />
        <property name="start" value="Thu Nov 27 23:55:32 GMT 2014" />
        <property name="test" value="vm/verifier/VerifyStackForExceptionHandlers.java" />
        <property name="user.name" value="root" />
        <property name="work" value="/wrkdirs/usr/ports/java/openjdk8/work/jtreg-work/jdk/vm/verifier" />
    </properties>
    <testcase classname="vm/verifier/VerifyStackForExceptionHandlers.java" name="VerifyStackForExceptionHandlers" time="0.392" >
    </testcase>
    <system-out>PASS
    </system-out>
    <system-err>
JavaTest Message:  Test complete.

    </system-err>
</testsuite>


> On Nov 25, 2014, at 12:23 PM, Brian Gardner <openjdk at getsnappy.com> wrote:
> 
> Hi Craig,
> I’ll continue the investigation, I’ve been following the openjdk8 posts from Kit.  I initially ported openjdk8 and was running jtreg tests as part of the process.  If I recall correctly the output was HTML but I think this is configurable, worse case I probably can convert it into supported xml format.  I’m thinking we can bake running tests into the openjdk8 port Makefile, and use poudriere testport and pass in a script that will upload the results into Jenkins.  That should make it really easy to run and maintain this on all supported platforms.  
> 
> Brian
> 
>> On Nov 25, 2014, at 12:11 PM, Craig Rodrigues <rodrigc at FreeBSD.org <mailto:rodrigc at FreeBSD.org>> wrote:
>> 
>> Hi,
>> 
>> Thanks, Brian!
>> 
>> Well, first of all, we need someone to figure out how to run the jtreg tests under FreeBSD,
>> and make that process reproducible.  Would you be willing to do the initial investigation for this?
>> If you look at Kip Macy's postings on the freebsd-java list in October, you can see that he
>> managed to get the tests to run, so hopefully Kip can answer any questions about what he did to get
>> the tests to run.
>> 
>> I'm not familiar with jtreg.  What ports are you thinking that we need to build with poudriere in order to run it?
>> 
>> Le me try to answer your questions:
>> 
>> 1)  Let's test on amd64 first, just to iron out all the issues.  Once we get that working, we can add more platforms, such as i386.
>> 2)  For an initial effort, maybe we can schedule the tests to run once a week, via the Jenkins job scheduler.  We can see if
>>      it makes sense to run the tests more or less frequently after that.
>> 3)  I don't know what format the jtreg test output produces.  If it can produce JUnit XML output, that would be ideal because
>>      it can be imported directly into Jenkins.  If it does not do that, then even if we make the text output of the test logs available online,
>>      that is a good start.
>> 
>> --
>> Craig
>> 
>> On Sun, Nov 23, 2014 at 9:15 PM, Brian Gardner <openjdk at getsnappy.com <mailto:openjdk at getsnappy.com>> wrote:
>> Hello,
>> I’d like to help.  Sounds like those scripts would be fairly easy to write.  Would it be a good idea to try and decouple building of the ports (using poudriere) from the actual running of the jtreg tests.  In this plan you’d have a separate job that builds and maintains an up-to-date repository of  packages for the supported platforms (10.0 amd64, 9.3 i386), then as part of running tests you’d run pkg to update the packages as the first step.
>> 
>> I have some questions:
>> 1) What platforms will be tested?
>> 2) How often will tests run / What will trigger test runs?
>> 3) How will results of the jtreg tests be presented on Jenkins?
>> 
>> Brian Gardner
>> 
>> 
>> In Response To:
>> 
>> Hi,
>> 
>> Recently Kip Macy tried to get the openjdk8 tests working
>> under FreeBSD.  He struggled a bit:
>> 
>> https://lists.freebsd.org/pipermail/freebsd-java/2014-October/010928.html <https://lists.freebsd.org/pipermail/freebsd-java/2014-October/010928.html>
>> https://lists.freebsd.org/pipermail/freebsd-java/2014-October/010929.html <https://lists.freebsd.org/pipermail/freebsd-java/2014-October/010929.html>
>> https://lists.freebsd.org/pipermail/freebsd-java/2014-October/010930.html <https://lists.freebsd.org/pipermail/freebsd-java/2014-October/010930.html>
>> 
>> I think it would be good to get these tests running under FreeBSD
>> continous integration.  These tests stress things like threading, memory
>> management, networking, and virtual memory.  Running these tests
>> would go a long way to improving Java support under FreeBSD,
>> because it would give a better idea of what can be fixed in FreeBSD itself.
>> 
>> I am a bit overstretched at the moment to take this on.
>> 
>> 
>> I need scripts that can do the following:
>> 
>> (1)  Download the necessary packages, openjdk8, jtreg, etc.
>> (2)  Apply necessary patches to packages in (1)
>> (3)  Build the packages in (1) after patches applied
>> (4)  Run the tests
>> 
>> I would also like:
>> (A)  Scripts should be committed to https://github.com/freebsd/freebsd-ci <https://github.com/freebsd/freebsd-ci>
>> (B)  Script should run under FreeBSD
>> (C)  Scripts should be easy to run under a Jenkins job at
>>        https://jenkins.freebsd.org <https://jenkins.freebsd.org/>
>> (D)  Followup status and discussion should happen on
>>        the freebsd-testing at freebsd.org <https://lists.freebsd.org/mailman/listinfo/freebsd-testing> mailing list
>> 
>> Is anyone interested in helping out with this?  Starting with some
>> of the things that Kip did would be a good place to start.
>> 
>> Thanks.
>> --
>> Craig
>> 
>> 
>> 
>>> On Nov 23, 2014, at 8:49 PM, Craig Rodrigues <rodrigc at FreeBSD.org <mailto:rodrigc at FreeBSD.org>> wrote:
>>> 
>>> Hi,
>>> 
>>> Read this:
>>> https://lists.freebsd.org/pipermail/freebsd-testing/2014-November/000668.html <https://lists.freebsd.org/pipermail/freebsd-testing/2014-November/000668.html>
>>> 
>>> and reply to that posting on freebsd-testing at freebsd.org <mailto:freebsd-testing at freebsd.org> (subscribe to that list if you
>>> are not on it) with the items that you think that you can
>>> complete.
>>> 
>>> --
>>> Craig
>>> 
>>> On Sun, Nov 23, 2014 at 8:28 PM, Brian Gardner <openjdk at getsnappy.com <mailto:openjdk at getsnappy.com>> wrote:
>>> Hi Craig,
>>> I would like to help.  Let me know what I can do.
>>> 
>>> Brian Gardner
>>> 
>>> > On Nov 22, 2014, at 4:40 PM, Craig Rodrigues <rodrigc at FreeBSD.org <mailto:rodrigc at FreeBSD.org>> wrote:
>>> >
>>> > FYI,
>>> >
>>> > https://lists.freebsd.org/pipermail/freebsd-testing/2014-November/000668.html <https://lists.freebsd.org/pipermail/freebsd-testing/2014-November/000668.html>
>>> >
>>> > Please send followups to freebsd-testing at freebsd.org <mailto:freebsd-testing at freebsd.org>.
>>> >
>>> > --
>>> > Craig
>>> > _______________________________________________
>>> > freebsd-java at freebsd.org <mailto:freebsd-java at freebsd.org> mailing list
>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-java <http://lists.freebsd.org/mailman/listinfo/freebsd-java>
>>> > To unsubscribe, send any mail to "freebsd-java-unsubscribe at freebsd.org <mailto:freebsd-java-unsubscribe at freebsd.org>"
>>> 
>>> 
>> 
>> 
>

More information about the freebsd-testing mailing list