[opennms-devel] OpenNMS 1.14+ on FreeBSD 8, 9, 10, 11

[Seth Leger]

Hi Paul,

To figure out what has changed in OpenNMS, it is easiest to read the
wiki and github docs:

http://www.opennms.org/wiki/What's_New_in_OpenNMS_1.10
http://www.opennms.org/wiki/What's_New_in_OpenNMS_1.12
https://github.com/OpenNMS/opennms/blob/release-14.0.0/WHATSNEW.md
https://github.com/OpenNMS/opennms/blob/release-15.0.2/WHATSNEW.md

However, I think you are looking at this the wrong way. If you look
through these changes, you will see at a high level what has been added
and removed but these feature changes correspond to tens of thousands of
lines of code changed in OpenNMS and hundreds of thousands of lines of
code changed if you include the cumulative changes in all of our
dependencies. Any ONE of those changes could be causing the negative
interactions you are seeing with a particular JVM on FreeBSD.

We have a large array of unit tests and integration tests that are run
dozens of times every week on Linux and many of our developers run in
OSX and if we find a JVM crash on those systems, we would look for a
solution immediately. However, I say "would" because I can't remember
the last time we encountered a JVM segfault crash on those platforms.

The most productive thing for you to do is, like Ben said, give us a
stack trace of the crashing JVM. Even better would be to set up an
environment where you could run the unit and integration tests
continuously like we do internally but run them on your FreeBSD
environments that you are concerned about. If your environment can run
the tests successfully many times without encountering JVM crashes, that
will give us extra confidence that the code can execute successfully on
your platform. Then you could start looking at particular runtime
conditions that could be causing the crashes.

http://www.opennms.org/wiki/Running_Unit_Tests

-- Seth



On 3/30/15 9:48 PM, Paul Pathiakis wrote:
> Seth,
>
> THANK YOU!
>
> I needed to know that.  However, when did this all change?  (I know, I
> could read the RELEASE NOTES etc, but it's easier is someone who knows
> just tells me. :-)  )  Here's the thing, I was trying to get OpenNMS
> 1.14 working with all the versions of JDK and they all failed on FreeBSD.
>
> I assume the advanced features continued to be implemented from 1.10
> to 1.14.  Now, I can at least get some idea of where to start.  I
> might be able to track this down to things that were implemented in a
> certain version from 1.10 to 1.14.
>
> I don't know how long I'll be able to use openjdk6 before having to go
> to openjdk7 on 1.1x, but at least it's a start. Like I said, I'm more
> than willing to help assist in this.
>
> Do you think that I should be able to use openjdk7 from 1.10 going
> forward?  Is JDK 1.7 backward compatible with everything in 1.6?  If
> so, I should just start 1.7 on OpenNMS 1.10 and go forward from
> there.  Is that a good plan?
>
> On 03/30/2015 19:42, Seth Leger wrote:
>> Hi Paul,
>>
>> One thing to keep in mind is that the JVM standard has changed
>> drastically since OpenNMS 1.10 was released. OpenNMS 1.10 running on
>> JDK 6 is a very different ecosystem than OpenNMS 15. OpenNMS 15 uses
>> a variety of modern JVM features that require us to use JDK 7. Our
>> dependencies exercise advanced JVM features such as dynamic proxy
>> generation and bytecode manipulation. We have an entire OSGi plugin
>> framework embedded inside our system now that didn't exist in OpenNMS
>> 1.10.
>>
>> For this reason, we rely on some of the newest, least-tested code in
>> the JVM in some cases. JVM code can and does have bugs that can cause
>> crashes. I ran into a case today where the JAXB implementation in
>> Oracle's JDK7 on Linux would not run some of our topology code correctly.
>>
>> As Ben noted, our code is written in Java and as far as we can tell,
>> the bytecode that is compiled by javac conforms properly to the Java
>> bytecode standards (as you would trust any compiler to do). If you
>> have valid bytecode, than any further crash in the JVM is, as Ben
>> noted, a JVM issue.
>>
>> -- Seth
>>
>>
>> On 3/30/15 6:02 PM, Paul Pathiakis wrote:
>>>
>>> On 03/30/2015 11:56, Benjamin Reed wrote:
>>>> On 3/28/15 3:06 PM, Paul Pathiakis wrote:
>>>>> Openjdk crashes almost 
>>>>> immediately while the linux java continues for a longer duration but 
>>>>> also crashes.
>>>> Unless it is crashing in jrrd or jicmp which we wrote, it is by
>>>> definition a JVM issue.  Unless we're running out of memory, it should
>>>> be impossible to put something in a jar that crashes the JVM, and if it
>>>> does, it's a JVM bug.  Since the Linux JDK doesn't crash under the Linux
>>>> kernel, that would mean it has to be some interaction in the FreeBSD
>>>> side breaking it.
>>>>
>>>> Without a crash log, though, it's hard to say.  :)
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
>>>> by Intel and developed in partnership with Slashdot Media, is your hub for all
>>>> things parallel software development, from weekly thought leadership blogs to
>>>> news, videos, case studies, tutorials and more. Take a look and join the 
>>>> conversation now. http://goparallel.sourceforge.net/
>>>>
>>>>
>>>> _______________________________________________
>>>> Please read the OpenNMS Mailing List FAQ:
>>>> http://www.opennms.org/index.php/Mailing_List_FAQ
>>>>
>>>> opennms-devel mailing list
>>>>
>>>> To *unsubscribe* or change your subscription options, see the bottom of this page:
>>>> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>>> Ben,
>>>
>>>     More than happy to provide any crash dumps.  However, I can't
>>> agree with it 'must be' the JVM.  Ron Roskens and I have delved into
>>> this and found some interesting things going on.  The behavior
>>> differs between the Linux JVM on FreeBSD linuxlator and the openjdk
>>> on FreeBSD. 
>>>     Once upon a time, previous OpenNMS versions always worked on
>>> FreeBSD on Java 1.4, 1.5, 1.6.  There was never an issue with
>>> getting it running, there just wasn't a package/port. 
>>>
>>> Here's one of the e-mails that I've received regarding this:
>>>
>>> *Hi Paul, good morning.***
>>>
>>> **
>>>
>>> ***I’ve been reading a lot of your posts to the lists about OpenNMS
>>> on FreeBSD.***
>>>
>>> **
>>>
>>> *I have been running OpenNMS v1.10.6 on FreeBSD 9 for years with no
>>> trouble, but decided to upgrade.  Heh.***
>>>
>>> **
>>>
>>> *See versions of software I’m running below.***
>>>
>>> **
>>>
>>> *I have the same symptoms you described last fall:  it runs for a
>>> while, then boom – JVM blows up.***
>>>
>>> **
>>>
>>> *Have you had any luck getting it working?***
>>>
>>> **
>>>
>>> *Thanks!***
>>>
>>> **
>>>
>>> *___________________________***
>>>
>>> **
>>>
>>> *Kris Jacobs***
>>>
>>> *Network Administrator**
>>> **% uname –a**
>>> **
>>> **FreeBSD NETMONv2.calhouncountymi.local 10.1-RELEASE FreeBSD
>>> 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014    
>>> root at releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64**
>>> **
>>> **% pkg version | grep jdk  **
>>> **
>>> **openjdk-7.76.13_1,1**
>>> **
>>> **opennms-15.0.1-1**
>>> *
>>>
>>> So, here's an example of someone running OpenNMS 1.10.6 on FreeBSD 9
>>> and openjdk6.  It ran "for years".  I had run FreeBSD 8.0 with
>>> OpenNMS 1.{5,6}(I think) with an unknown JDK version and it ran for
>>> years without issue.  Again, what has happened?  The user, Kris,
>>> (above) has tried getting it to work on OpenNMS 1.15 with openjdk
>>> 6,7,8.  None of it works.  I believe OpenNMS to be the best NMS
>>> system out there right  now.  However, I'm using Zabbix for all of
>>> my new installs which seems to be my only alternative at this point. 
>>>
>>> Could someone tell me if this is working on OSX reliably?  That may
>>> be an alternative.  BTW, I don't have any other issues with other
>>> apps that are using openjdk crashing on FreeBSD, it seems to be
>>> OpenNMS is the only app with a problem.
>>>
>>> I'm also sending this to the java maillist at FreeBSD.  I'm hoping
>>> that the two groups can work together to resolve the issue. 
>>>
>>> My being in the middle may be hampering the issue but what seems to
>>> be hurting more is the fact that there doesn't seem to be anyone
>>> willing to work the issue to PROVE it is something having to do with
>>> the JDK.  People stating "Well, it works on Linux, with the Linux
>>> JDK, which we have Linux specific conditionals set in XML code and
>>> it works fine" comes across as a serious dodge.  (Ron Roskens
>>> pointed out a couple of issues with the code using specific 'epoll'
>>> calls.  Kudos to him for digging in and finding that.)
>>>
>>> Seriously, if OpenNMS says "We will not support OpenNMS on FreeBSD",
>>> then please make the statement for everyone to see so they know to
>>> stop using and supporting this product.  Just close the door and
>>> well meaning people like myself will just walk away. Sevan used to
>>> complain that the upstream support in the OpenNMS group was what
>>> kept the product from being stable on more platforms, I'm starting
>>> to lean that way as well.  (With Ron Roskens being the exception, of
>>> course.)  (Sad considering that Juniper uses FreeBSD for JunOS and
>>> Playstation is built on it, and Apple is built on it.... etc)
>>>
>>> Something that the OpenNMS project should consider:  software being
>>> well-supported on more than just one or two platforms is the best
>>> way to hedge your bets.  Right now, there is serious slippage on a
>>> couple of platforms in the industry.  Any software project that has
>>> all their eggs in one or two baskets could end up blowing away if
>>> those eggs lose market share.
>>>
>>> Tarus, can you weigh in on this?
>>>
>>> BTW, where can I get the source for 10.x?  I'd like to see if it
>>> still runs on openjdk6.  That might be a solid place to start on my
>>> end.  If that can work, I'll try openjdk 7.
>>>
>>> Thank you,
>>>
>>> Paul Pathiakis
>>>
>>>
>>>