Tomcat Debugging (OT most likely)

[Paul Procacci]

Hey all,

This question is probably best suited for tomcat-users, but being this 
is an a FBSD machine I'd figure I'd ask here first.  I have a client 
running apache w/ tomcat serving thousands of requests per second.  It 
works, but with one caveat, pages load extremely slowly.  When I first 
saw the request, I figured this client was running into some kind of 
resource bottleneck, but that's where my problems started.  I'm unable 
to find any resource that is being starved.  I turn to `the mighty list` 
for guidance.

This particular machine is running FreeBSD 6.2.  It has a bce network 
card that currently connected to a GigE switch with a 10G uplink.  Even 
with all this available bandwidth, this client's application *currently* 
is not doing any more than 50Mb/sec.  ::  Rule out network congestion.

Moving along to the system ram :: 2 Gigs are in the machine, with very 
little usage:
Mem: 724M Active, 874M Inact, 338M Wired, 64M Cache, 112M Buf, 4944K Free

A dual core cpu is in the machine, again with very little usage:
CPU states:  8.6% user,  0.0% nice,  3.9% system,  2.8% interrupt, 84.7% 
idle

Disk throughput is negligable at this time (50KB/sec :: 3tps).

I did start off mentioning this machine uses tomcat...so let me 
continue.  We've setup the java process to use libthr via libmap.conf:
#############
cat /etc/libmap.conf
[java]
libpthread.so.2 libthr.so.2
libpthread.so libthr.so
#############

And added the following to rc.conf::
###############
tomcat55_java_opts="-XX:MaxPermSize=512m -Xmx512M"
###############

Apache Configuration:
###################
MaxClients         1024
###################

Tomcat Configuration:
###################
maxThreads="1200"
###################

----------------------------------------------------------------
So onto my question...the slowness being encountered.  My initial 
thought was the slowness was due to quite possibly the cpu being 
hammered by numerous requests.  After checking however that's not the 
case, then I looked at the networking equipment with my networking team, 
that's ok too.  (I think) That leaves a) thread contention somewhere  b) 
apache misconfiguration and c) tomcat configuration.

Both myself and 2 other admins have looked over the apache configuration 
and tomcat configuration and we believe that side of things is probably 
ok.  That's leaves weird contention in the kernel or userland mutexes or 
something along those lines.

Here is our current connection count on the external interface ::
################################
sockstat -4cp 80 | wc -l
     997
################################

Here is our connection count on localhost (from apache to tomcat process 
and vica versa) ::
################################
sockstat -4cp 8009  | wc -l
    1679
################################
----------------------------------------------------------------

And finally.....top output:

last pid: 57747;  load averages:  0.79,  1.12,  
0.87                                                                                
up 153+17:56:53 18:36:57
<#--snip -->
  PID USERNAME   PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
57460 www         96    0  1385M   432M ucond  0   2:44  2.88% java
57460 www         96    0  1385M   432M ucond  0   2:44  2.73% java
57460 www         96    0  1385M   432M select 0   2:44  0.15% java
57460 www         96    0  1385M   432M select 1   2:44  0.15% java
<#--snip-->

The above lines repeat up to the number of threads we have.  `states` 
are in ucond and select.
----------------------------------------------------------------

Hopefully I hven't bored anyone....yet, and would appreciate any 
guildance.  Maybe good tools for debugged kernel threads, or a simple 
slap in the face will do nicely.

Thanks in advance,
Paul