java crashing when load average is high
kwsn at earthlink.net
Wed Apr 20 23:10:10 PDT 2005
On Wed, 2005-04-20 at 08:59 -0600, Greg Lewis wrote:
> On Sun, Apr 17, 2005 at 02:06:24PM -0500, Toby Murray wrote:
> > I am working on getting a new server online. It is a Dell Poweredge
> > with dual HT Xeons and 1GB of RAM. We are running FreeBSD 5.4-RC1 and
> > java 1.4.2_08.
> > java version "1.4.2_08"
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_08-b03)
> > Java HotSpot(TM) Client VM (build 1.4.2_08-b03, mixed mode)
> > I have a java program that reads XML files over an HTTP stream and
> > inserts data from them into a mysql database. After getting java up
> > and running everything appeared to be working normally at first and
> > the program ran fine. But then I tried running it while something
> > else was compiling and the load average was up around 5 or 6.
> > Suddenly the behavior of the JVM became extremely erratic.
> > Sometimes the whole JVM dumps with a signal 11 and leaves the following in dmsg:
> > kernel trap 12 with interrupts disabled
> > Other times the program throws random exceptions. I have seen 5 or 6
> > different exceptions being thrown at seemingly random places in the
> > code while parsing the same data.
> > A third behavior is for java to go into what appears to be an infinite
> > loop somewhere. Usually when my program runs, there is one java
> > thread and one mysql process that each take up ~70% of one CPU. When
> > things start looping, 2 java processes start sucking up as much CPU as
> > they can and mysql drops to nothing. This continues until I pull out
> > the kill -9.
> > So to summarize it all:
> > java works fine if it is the only thing running
> > If the box is already under load, java malfunctions in one of 3 ways:
> > Signal 11, random Exception thrown or it starts looping without
> > getting anything done.
> > Any clues? I see several other threads on the list about signal 11
> > problems but I didn't see anything about the problems being related to
> > load. Let me know if you need more info on something.
> Sounds like you are running into a race condition or corruption problem.
> There are several places this could be, the most likely being the Linux
> emulation layer.
> You may be able to use the native 1.4 if its compiled on an x86, but I'm
> not sure how well that works on amd64. If you were particularly brave
> you could try the native 1.5 which will compile natively on amd64.
Native x86 1.4 went straight into the loop. Native x86 1.5 immediately
segfaulted - even on `java -version`. I removed all java-related ports
and linux ports, then reinstalled linux_base-8 and linux-sun-jdk14 and
tried to build jdk15 again. That build hung. I tried it again this
afternoon and somehow, it succeeded (after manual intervention when the
build craps out on a spectacularly long command, but I expected that).
We tested it this evening, and it's stable under load. So now we've got
java working on amd64 with our Xeons. For anyone else trying to use
jdk15 on EM64T Xeons, be sure to use the patch in the message linked
below, otherwise your javavm will abort with an illegal instruction
Thanks to Ernst Winter for all of his help in trying to get java running
The patch to enable jdk15 to run on EM64T Xeons:
More information about the freebsd-java