crash of diablo-amd64 in production system

Alumnivereniging Nederlands mail at hebbanollavogala.nl
Thu Jul 5 09:39:25 UTC 2007


Hi list,

I have had a crash of the diablo JVM on an amd64 production server.
The JVM is running 10 web apps in Tomcat.

The GC log was overwritten at the (rush) restart of the JVM. Is it 
possible to prevent this?

I have observed that the free memory reported by the JVM in Tomcat 
manager decreases gradually over time.

What also seems strange to me: in the GC logs I collected there had not 
been a full GC over a period of two weeks. Only 'scavenges' are 
performed. I am using HP Jmeter to analyze the GC logs.

These are my JVM options. Can you comment on them?

jvm_args: -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -verbose:gc 
-XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps 
-Xloggc:/var/log/tomcat/gc.log -Djava.endorsed.dirs= 
-Dcatalina.base=/usr/local/tomcat5.5 
-Dcatalina.home=/usr/local/tomcat5.5 
-Djava.io.tmpdir=/usr/local/tomcat5.5/temp

The server has 4GB of memory and is mainly running Tomcat5.5.20 on 
diablo-1.5.0-b01  and Apache2.2.3.


I have the hs_err crash log attached. I would be very grateful if 
someone can point to a possible cause of the problem.

Thank you,
Neil
-------------- next part --------------
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0x0000000800ed3399, pid=28387, tid=0x6c8800
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (diablo-1.5.0_07-b01 mixed mode)
# Problematic frame:
# V  [libjvm.so+0x43f399]
#

---------------  T H R E A D  ---------------

Current thread (0x00000000006c3c00):  JavaThread "CompilerThread0" daemon [_thread_in_vm, id=7112704]

siginfo:si_signo=11, si_errno=0, si_code=12, si_addr=0x0000000000000010

Registers:
RAX=0x0000000000000000, RBX=0x0000000000000000, RCX=0x0000000000000108, RDX=0x00000000035b84e0
RSP=0x00007fffff2f5420, RBP=0x00007fffff2f5450, RSI=0x00000008012a64cb, RDI=0x000000080142af80
R8 =0x0000000000000160, R9 =0x0000000000000002, R10=0x000000000051a7f0, R11=0x00007fffff2f62c8
R12=0x00000000035b84e0, R13=0x00000000035b85f0, R14=0x0000000000000002, R15=0x0000000000000160
RIP=0x0000000800ed3399, EFL=0xffffffff80546320, CSGSFS=0xffffff0085ebb260, ERR=0xffffff0045c0e9b0
  TRAPNO=0x0000000000000000

Top of Stack: (sp=0x00007fffff2f5420)
0x00007fffff2f5420:   000000000050c580 0000000000000000
0x00007fffff2f5430:   00000000035b84e0 00000000035b85f0
0x00007fffff2f5440:   0000000000000002 0000000000000160
0x00007fffff2f5450:   00007fffff2f54f0 0000000801170bc6
0x00007fffff2f5460:   00000000035b85f0 0000000000000000
0x00007fffff2f5470:   000000000050cca8 00000001006c8800
0x00007fffff2f5480:   000000000051a7f0 00007fffff2f62c8
0x00007fffff2f5490:   00007fffff2f62b0 00000000006c8070
0x00007fffff2f54a0:   0000000000000000 0000001000f69711
0x00007fffff2f54b0:   0000000000000010 ffffffff00000000
0x00007fffff2f54c0:   000000080b7b2938 0000000000000160
0x00007fffff2f54d0:   0000000000000000 00000000006c8070
0x00007fffff2f54e0:   000000080b7b2938 0000000000000010
0x00007fffff2f54f0:   00007fffff2f55b0 0000000801170f2a
0x00007fffff2f5500:   0000000000000010 0000000100000010
0x00007fffff2f5510:   0000000000000000 00000000006c8070
0x00007fffff2f5520:   00000000035b84e0 00007fff00000002
0x00007fffff2f5530:   00000000035b85f0 00007fffff2f62b0
0x00007fffff2f5540:   00007fffff2f62c8 0000000000000000
0x00007fffff2f5550:   000000000051a7f0 0000000800e9c51b
0x00007fffff2f5560:   0000000000000010 ffffffff00000000
0x00007fffff2f5570:   000000000050b480 0000000800763ef5
0x00007fffff2f5580:   00007fffff2f55b0 000000000051a7f0
0x00007fffff2f5590:   00000000006e3618 0000000003560550
0x00007fffff2f55a0:   00000000006c3c00 00007fffff2f6ad0
0x00007fffff2f55b0:   00007fffff2f56b0 0000000800e9df35
0x00007fffff2f55c0:   00007fff00000000 00000000006c8070
0x00007fffff2f55d0:   00000000035b84e0 0000000800000002
0x00007fffff2f55e0:   00000000035b85f0 00007fffff2f62b0
0x00007fffff2f55f0:   00007fffff2f62c8 0000000000000000
0x00007fffff2f5600:   000000000051a7f0 0000000000000000
0x00007fffff2f5610:   0000004600000046 0000004600000046 

Instructions: (pc=0x0000000800ed3399)
0x0000000800ed3389:   d4 4c 89 7d f8 48 83 ec 30 48 8b 3d 2f ce 62 00
0x0000000800ed3399:   44 89 43 10 89 4b 14 45 89 cf 48 89 73 08 41 89 

Stack: [0x00007fffff1f7000,0x00007fffff2f7000),  sp=0x00007fffff2f5420,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x43f399]
V  [libjvm.so+0x6dcbc6]
V  [libjvm.so+0x6dcf2a]
V  [libjvm.so+0x409f35]
V  [libjvm.so+0x4531e8]
V  [libjvm.so+0x3f09e0]
V  [libjvm.so+0x457dec]
V  [libjvm.so+0x458304]
V  [libjvm.so+0x7a762d]
V  [libjvm.so+0x6f06d2]


Current CompileTask:
opto:8729      antlr.CommonAST.setType(I)V (6 bytes)


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x0000000002fd1c00 JavaThread "File Reaper" daemon [_thread_blocked, id=21490688]
  0x0000000003159000 JavaThread "Scheduled tasks handler for app1" daemon [_thread_blocked, id=21810176]
  0x0000000003007000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9" daemon [_thread_blocked, id=21695488]
  0x0000000002f77c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#8" daemon [_thread_blocked, id=21306368]
  0x0000000002f5f400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7" daemon [_thread_blocked, id=14132224]
  0x0000000002f14c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6" daemon [_thread_blocked, id=44764160]
  0x0000000001eec800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5" daemon [_thread_blocked, id=15647744]
  0x00000000013f8800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4" daemon [_thread_blocked, id=45743104]
  0x0000000001ea8c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3" daemon [_thread_blocked, id=13465600]
  0x00000000014cc400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=46226432]
  0x00000000014e0c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=45820928]
  0x00000000014e0000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=45116416]
  0x0000000001451800 JavaThread "Timer-15" daemon [_thread_blocked, id=45874176]
  0x00000000014ae000 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=45898752]
  0x0000000002db8800 JavaThread "File Reaper" daemon [_thread_blocked, id=45423616]
  0x0000000002c47400 JavaThread "TP-Processor16" daemon [_thread_blocked, id=46017536]
  0x0000000002c45800 JavaThread "TP-Processor15" daemon [_thread_blocked, id=44719104]
  0x0000000002afbc00 JavaThread "TP-Processor14" daemon [_thread_blocked, id=44566528]
  0x0000000002c4e800 JavaThread "TP-Processor13" daemon [_thread_blocked, id=36605952]
  0x0000000002afb000 JavaThread "File Reaper" daemon [_thread_blocked, id=44556288]
  0x00000000027ee800 JavaThread "File Reaper" daemon [_thread_blocked, id=14202880]
  0x00000000028c8c00 JavaThread "Scheduled tasks handler for app2" daemon [_thread_blocked, id=13338624]
  0x0000000002a6b800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9" daemon [_thread_blocked, id=44532736]
  0x0000000002a58800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#8" daemon [_thread_blocked, id=44479488]
  0x0000000002a31800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7" daemon [_thread_blocked, id=44401664]
  0x0000000002a2f800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6" daemon [_thread_blocked, id=44241920]
  0x0000000002a15800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5" daemon [_thread_blocked, id=44233728]
  0x0000000002a0f800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4" daemon [_thread_blocked, id=44127232]
  0x0000000002a00800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3" daemon [_thread_blocked, id=44102656]
  0x00000000029fa800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=44041216]
  0x00000000029b9c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=44016640]
  0x0000000002813800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=43751424]
  0x00000000028fdc00 JavaThread "Timer-13" daemon [_thread_blocked, id=42021888]
  0x0000000002768400 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=39359488]
  0x0000000002589c00 JavaThread "File Reaper" daemon [_thread_blocked, id=39366656]
  0x000000000247ec00 JavaThread "Scheduled tasks handler for app3" daemon [_thread_blocked, id=13506560]
  0x0000000002335c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9" daemon [_thread_blocked, id=13121536]
  0x00000000022e8c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#8" daemon [_thread_blocked, id=13514752]
  0x0000000000b9f400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7" daemon [_thread_blocked, id=12187648]
  0x0000000000b18c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6" daemon [_thread_blocked, id=13694976]
  0x000000000105b800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5" daemon [_thread_blocked, id=13674496]
  0x0000000000d0a400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4" daemon [_thread_blocked, id=12185600]
  0x00000000014df800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3" daemon [_thread_blocked, id=13119488]
  0x0000000000e16c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=15606784]
  0x0000000000df0c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=23683072]
  0x0000000000df0800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=19132416]
  0x0000000000ce3000 JavaThread "Timer-12" daemon [_thread_blocked, id=20247552]
  0x0000000000d8bc00 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=14771200]
  0x00000000024c8400 JavaThread "Scheduled tasks handler for app4" daemon [_thread_blocked, id=15168512]
  0x0000000002243800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9" daemon [_thread_blocked, id=13255680]
  0x0000000002061400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#8" daemon [_thread_blocked, id=12475392]
  0x0000000001767c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7" daemon [_thread_blocked, id=13257728]
  0x0000000001a50400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6" daemon [_thread_blocked, id=14984192]
  0x0000000000d19c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5" daemon [_thread_blocked, id=15410176]
  0x000000000197fc00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4" daemon [_thread_blocked, id=13734912]
  0x00000000013c0400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3" daemon [_thread_blocked, id=12473344]
  0x000000000134f000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=14615552]
  0x0000000000eb2c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=24075264]
  0x0000000000eb2800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=21777408]
  0x0000000000eb2000 JavaThread "Timer-11" daemon [_thread_blocked, id=27375616]
  0x0000000000ef9800 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=24064000]
  0x0000000002212800 JavaThread "TP-Processor12" daemon [_thread_in_native, id=35160064]
  0x00000000021df000 JavaThread "TP-Processor11" daemon [_thread_in_Java, id=28057600]
  0x00000000021dec00 JavaThread "TP-Processor10" daemon [_thread_blocked, id=22144000]
  0x00000000021de400 JavaThread "TP-Processor9" daemon [_thread_in_native, id=29719552]
  0x000000000141d400 JavaThread "File Reaper" daemon [_thread_blocked, id=19965952]
  0x0000000001db5400 JavaThread "File Reaper" daemon [_thread_blocked, id=31152128]
  0x0000000001ce2000 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=30286848]
  0x000000000134c400 JavaThread "File Reaper" daemon [_thread_blocked, id=28303360]
  0x00000000013f0800 JavaThread "TP-Processor8" daemon [_thread_in_native, id=23311360]
  0x000000000115cc00 JavaThread "TP-Processor7" daemon [_thread_in_native, id=20907008]
  0x000000000115c400 JavaThread "TP-Processor6" daemon [_thread_in_native, id=18204672]
  0x00000000013e3c00 JavaThread "TP-Processor5" daemon [_thread_in_native, id=18202624]
  0x0000000001fb4800 JavaThread "TP-Monitor" daemon [_thread_blocked, id=33260544]
  0x0000000001fae800 JavaThread "TP-Processor4" daemon [_thread_in_native, id=33244160]
  0x0000000001faa800 JavaThread "TP-Processor3" daemon [_thread_blocked, id=33219584]
  0x000000000082ec00 JavaThread "TP-Processor2" daemon [_thread_blocked, id=33203200]
  0x000000000082e400 JavaThread "TP-Processor1" daemon [_thread_in_native, id=8579072]
  0x0000000001f8e400 JavaThread "http-9442-Monitor" [_thread_blocked, id=33116160]
  0x0000000001f89400 JavaThread "http-9442-Processor25" daemon [_thread_in_native, id=33087488]
  0x0000000001e73400 JavaThread "http-9442-Processor24" daemon [_thread_blocked, id=33067008]
  0x0000000001df9400 JavaThread "http-9442-Processor23" daemon [_thread_blocked, id=31928320]
  0x0000000001d43400 JavaThread "http-9442-Processor22" daemon [_thread_blocked, id=31428608]
  0x0000000001d20400 JavaThread "http-9442-Processor21" daemon [_thread_blocked, id=30683136]
  0x0000000001cc1400 JavaThread "http-9442-Processor20" daemon [_thread_blocked, id=30539776]
  0x0000000001cb7400 JavaThread "http-9442-Processor19" daemon [_thread_blocked, id=30150656]
  0x0000000001cb0400 JavaThread "http-9442-Processor18" daemon [_thread_blocked, id=30109696]
  0x0000000001caa400 JavaThread "http-9442-Processor17" daemon [_thread_blocked, id=30081024]
  0x0000000001ca5400 JavaThread "http-9442-Processor16" daemon [_thread_blocked, id=30056448]
  0x0000000001ca0400 JavaThread "http-9442-Processor15" daemon [_thread_blocked, id=30035968]
  0x0000000001c99400 JavaThread "http-9442-Processor14" daemon [_thread_blocked, id=30015488]
  0x0000000001c94400 JavaThread "http-9442-Processor13" daemon [_thread_blocked, id=29986816]
  0x0000000001c86400 JavaThread "http-9442-Processor12" daemon [_thread_blocked, id=29966336]
  0x0000000001c81400 JavaThread "http-9442-Processor11" daemon [_thread_blocked, id=29908992]
  0x0000000001c7a400 JavaThread "http-9442-Processor10" daemon [_thread_blocked, id=29888512]
  0x0000000001c74400 JavaThread "http-9442-Processor9" daemon [_thread_blocked, id=29859840]
  0x0000000001c6f400 JavaThread "http-9442-Processor8" daemon [_thread_blocked, id=29835264]
  0x0000000001c5d400 JavaThread "http-9442-Processor7" daemon [_thread_blocked, id=29814784]
  0x0000000001bd2400 JavaThread "http-9442-Processor6" daemon [_thread_blocked, id=29741056]
  0x0000000001bcc400 JavaThread "http-9442-Processor5" daemon [_thread_blocked, id=29171712]
  0x0000000001b7c400 JavaThread "http-9442-Processor4" daemon [_thread_blocked, id=29147136]
  0x0000000001b4e800 JavaThread "http-9442-Processor3" daemon [_thread_blocked, id=28819456]
  0x0000000001b4e000 JavaThread "http-9442-Processor2" daemon [_thread_blocked, id=28632064]
  0x00000000017ac800 JavaThread "http-9442-Processor1" daemon [_thread_blocked, id=24824832]
  0x00000000017ab400 JavaThread "http-9441-Monitor" [_thread_blocked, id=24821760]
  0x0000000001488400 JavaThread "http-9441-Processor25" daemon [_thread_in_native, id=24817664]
  0x000000000127e400 JavaThread "http-9441-Processor24" daemon [_thread_blocked, id=21528576]
  0x0000000001175400 JavaThread "http-9441-Processor23" daemon [_thread_blocked, id=19390464]
  0x0000000000fa7400 JavaThread "http-9441-Processor22" daemon [_thread_blocked, id=18305024]
  0x0000000000bdb400 JavaThread "http-9441-Processor21" daemon [_thread_blocked, id=16412672]
  0x0000000000bda400 JavaThread "http-9441-Processor20" daemon [_thread_blocked, id=12431360]
  0x0000000000b26400 JavaThread "http-9441-Processor19" daemon [_thread_blocked, id=12427264]
  0x0000000000a13400 JavaThread "http-9441-Processor18" daemon [_thread_blocked, id=11689984]
  0x0000000000a12400 JavaThread "http-9441-Processor17" daemon [_thread_blocked, id=10563584]
  0x0000000000858400 JavaThread "http-9441-Processor16" daemon [_thread_blocked, id=10559488]
  0x0000000000843400 JavaThread "http-9441-Processor15" daemon [_thread_blocked, id=8749056]
  0x0000000000842400 JavaThread "http-9441-Processor14" daemon [_thread_blocked, id=8663040]
  0x000000000083d400 JavaThread "http-9441-Processor13" daemon [_thread_blocked, id=8658944]
  0x000000000083c400 JavaThread "http-9441-Processor12" daemon [_thread_blocked, id=8638464]
  0x0000000000839400 JavaThread "http-9441-Processor11" daemon [_thread_blocked, id=8634368]
  0x0000000000838400 JavaThread "http-9441-Processor10" daemon [_thread_blocked, id=8622080]
  0x000000000080d400 JavaThread "http-9441-Processor9" daemon [_thread_blocked, id=8617984]
  0x000000000080c400 JavaThread "http-9441-Processor8" daemon [_thread_blocked, id=8441856]
  0x0000000000801400 JavaThread "http-9441-Processor7" daemon [_thread_blocked, id=8437760]
  0x00000000007c9400 JavaThread "http-9441-Processor6" daemon [_thread_blocked, id=8392704]
  0x00000000007c8400 JavaThread "http-9441-Processor5" daemon [_thread_blocked, id=8163328]
  0x000000000075f400 JavaThread "http-9441-Processor4" daemon [_thread_blocked, id=8159232]
  0x0000000001573c00 JavaThread "http-9441-Processor3" daemon [_thread_blocked, id=7729152]
  0x0000000001191400 JavaThread "http-9441-Processor2" daemon [_thread_blocked, id=21071872]
  0x0000000000f47000 JavaThread "http-9441-Processor1" daemon [_thread_blocked, id=16020480]
  0x0000000000d70c00 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=15296512]
  0x0000000001675000 JavaThread "DateTimeListener for app5" [_thread_blocked, id=23389184]
  0x0000000001201c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=21753856]
  0x0000000001201400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=18880512]
  0x0000000000e22c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=18878464]
  0x0000000000e22400 JavaThread "Timer-10" daemon [_thread_blocked, id=14821376]
  0x0000000001b60800 JavaThread "DateTimeListener for app6" [_thread_blocked, id=28707840]
  0x00000000017cf400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=24967168]
  0x00000000012c4c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=24965120]
  0x00000000012c4400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=19679232]
  0x00000000015ff000 JavaThread "Timer-9" daemon [_thread_blocked, id=19677184]
  0x00000000018aa400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=25864192]
  0x000000000123fc00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=25862144]
  0x000000000123f400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=19134464]
  0x00000000010b6c00 JavaThread "DateTimeListener for app6" [_thread_blocked, id=17547264]
  0x0000000001173400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=18298880]
  0x0000000000bddc00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=18296832]
  0x0000000000bdd400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=12441600]
  0x0000000000a1ec00 JavaThread "Timer-7" daemon [_thread_blocked, id=12439552]
  0x0000000001913800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=26295296]
  0x0000000001913000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=26293248]
  0x000000000134f800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=20249600]
  0x000000000155f400 JavaThread "DateTimeListener for app8" [_thread_blocked, id=22411264]
  0x000000000148bc00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=24129536]
  0x000000000148b400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=21542912]
  0x000000000169ec00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=21540864]
  0x000000000169e400 JavaThread "Timer-5" daemon [_thread_blocked, id=23717888]
  0x0000000000b18400 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=11634688]
  0x0000000000d63000 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=14038016]
  0x00000000011a0800 JavaThread "Scheduled tasks handler for app9" [_thread_blocked, id=18484224]
  0x00000000012b4c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9" daemon [_thread_blocked, id=19994624]
  0x0000000001289c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#8" daemon [_thread_blocked, id=19613696]
  0x00000000011d7c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7" daemon [_thread_blocked, id=19437568]
  0x000000000116fc00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6" daemon [_thread_blocked, id=18708480]
  0x000000000113ac00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5" daemon [_thread_blocked, id=18282496]
  0x00000000010afc00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4" daemon [_thread_blocked, id=18065408]
  0x000000000100dc00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3" daemon [_thread_blocked, id=17496064]
  0x000000000100d400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=16832512]
  0x0000000000dd8c00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=16830464]
  0x0000000000dd8400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=14518272]
  0x0000000000a11800 JavaThread "Timer-2" daemon [_thread_blocked, id=14516224]
  0x0000000000d83400 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=14170112]
  0x0000000000d12400 JavaThread "Scheduled tasks handler for app10" [_thread_blocked, id=16837632]
  0x0000000000962000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9" daemon [_thread_blocked, id=9841664]
  0x000000000095c000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#8" daemon [_thread_blocked, id=9817088]
  0x000000000094f000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7" daemon [_thread_blocked, id=9763840]
  0x0000000000903000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6" daemon [_thread_blocked, id=9452544]
  0x00000000008ee000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5" daemon [_thread_blocked, id=9366528]
  0x00000000008e9000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4" daemon [_thread_blocked, id=9346048]
  0x00000000008e2000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3" daemon [_thread_blocked, id=9317376]
  0x000000000086b000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=8829952]
  0x0000000000ef9000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=13351936]
  0x0000000000ef8400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=15698944]
  0x00000000007d7000 JavaThread "Timer-1" daemon [_thread_blocked, id=15695872]
  0x0000000000b34c00 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=12840960]
  0x00000000007aa800 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=8039424]
  0x00000000006cfc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=7161856]
  0x00000000006c8c00 JavaThread "CompilerThread1" daemon [_thread_blocked, id=7141376]
=>0x00000000006c3c00 JavaThread "CompilerThread0" daemon [_thread_in_vm, id=7112704]
  0x00000000006bdc00 JavaThread "AdapterThread" daemon [_thread_blocked, id=7092224]
  0x00000000006a5c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=7067648]
  0x00000000006a5000 JavaThread "Finalizer" daemon [_thread_blocked, id=6969344]
  0x000000000062a800 JavaThread "Reference Handler" daemon [_thread_blocked, id=6466560]
  0x0000000000516800 JavaThread "main" [_thread_in_native, id=5332992]

Other Threads:
  0x000000000065f400 VMThread [id=6464512]
  0x0000000000523a00 WatcherThread [id=7190528]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x000000000050b480/0x000000000050cc80] CodeCache_lock - owner thread: 0x00000000006c3c00
[0x0000000000521e00/0x0000000000521e80] MethodCompileQueue_lock - owner thread: 0x00000000006c3c00

Heap
 PSYoungGen      total 247424K, used 123481K [0x000000087d130000, 0x000000088d180000, 0x00000008a7bd0000)
  eden space 237248K, 50% used [0x000000087d130000,0x000000088457b808,0x000000088b8e0000)
  from space 10176K, 43% used [0x000000088c790000,0x000000088cbdad90,0x000000088d180000)
  to   space 12608K, 0% used [0x000000088b8e0000,0x000000088b8e0000,0x000000088c530000)
 PSOldGen        total 1398144K, used 1095664K [0x0000000827bd0000, 0x000000087d130000, 0x000000087d130000)
  object space 1398144K, 78% used [0x0000000827bd0000,0x000000086a9cc3b0,0x000000087d130000)
 PSPermGen       total 415616K, used 415584K [0x0000000807bd0000, 0x00000008211b0000, 0x0000000827bd0000)
  object space 415616K, 99% used [0x0000000807bd0000,0x00000008211a8048,0x00000008211b0000)

Dynamic libraries:
0x0000000000400000 	/usr/local/diablo-jdk1.5.0/bin/java
0x000000080063d000 	/lib/libz.so.3
0x0000000800751000 	/lib/libpthread.so.2
0x000000080087c000 	/lib/libc.so.6
0x0000000800a94000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/server/libjvm.so
0x0000000801521000 	/usr/lib/libstdc++.so.5
0x0000000801720000 	/lib/libm.so.4
0x0000000801840000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/native_threads/libhpi.so
0x000000080194c000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libverify.so
0x0000000801a5b000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libjava.so
0x0000000801b83000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libzip.so
0x00000008a8ff4000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libnet.so
0x00000008aa536000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libmanagement.so
0x00000008aac38000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libnio.so
0x00000008aa001000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libcmm.so
0x00000008aa193000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libjpeg.so
0x00000008aa789000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libawt.so
0x00000008aa952000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libmlib_image.so
0x00000008aab04000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/headless/libmawt.so
0x00000008aad3f000 	/usr/local/diablo-jdk1.5.0/jre/lib/amd64/libfontmanager.so
0x000000080050a000 	/libexec/ld-elf.so.1

VM Arguments:
jvm_args: -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -verbose:gc -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Xloggc:/var/log/tomcat/gc.log -Djava.endorsed.dirs= -Dcatalina.base=/usr/local/tomcat5.5 -Dcatalina.home=/usr/local/tomcat5.5 -Djava.io.tmpdir=/usr/local/tomcat5.5/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/local/diablo-jdk1.5.0
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/home/user/bin
LD_LIBRARY_PATH=/usr/local/diablo-jdk1.5.0/jre/lib/amd64/server:/usr/local/diablo-jdk1.5.0/jre/lib/amd64:/usr/local/diablo-jdk1.5.0/jre/../lib/amd64
SHELL=/usr/local/bin/zsh
HOSTTYPE=FreeBSD
OSTYPE=FreeBSD
MACHTYPE=unknown

Signal Handlers:
SIGSEGV: [libjvm.so+0x7de650], sa_mask[0]=0xffffffff, sa_flags=0x00000002
SIGBUS: [libjvm.so+0x7de650], sa_mask[0]=0xffffffff, sa_flags=0x00000002
SIGFPE: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
SIGPIPE: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
SIGILL: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
SIGUSR1: [libjvm.so+0x6f30c0], sa_mask[0]=0x00000000, sa_flags=0x00000040
SIGUSR2: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042
SIGHUP: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002
SIGINT: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002
SIGQUIT: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002
SIGTERM: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002


---------------  S Y S T E M  ---------------

OS:FreeBSD
uname:FreeBSD 6.2-RC1 FreeBSD 6.2-RC1 #0: Mon Nov 20 14:05:41 CET 2006     root at server1.com:/usr/obj/usr/src/sys/AMD64 amd64
rlimit: STACK 8192k, CORE 0k, NOFILE 11095
CPU:total 2 em64t ht

Memory: 4k page, physical 3907036k

vm_info: Java HotSpot(TM) 64-Bit Server VM (diablo-1.5.0_07-b01) for freebsd-amd64, built on Sep 24 2006 16:09:01 by root with gcc 3.4.4 [FreeBSD] 20050518



More information about the freebsd-java mailing list