Tomcat Debugging (OT most likely)

[Paul Procacci]

Paul Procacci wrote:
> Hey all,
>
> This question is probably best suited for tomcat-users, but being this 
> is an a FBSD machine I'd figure I'd ask here first.  I have a client 
> running apache w/ tomcat serving thousands of requests per second.  It 
> works, but with one caveat, pages load extremely slowly.  When I first 
> saw the request, I figured this client was running into some kind of 
> resource bottleneck, but that's where my problems started.  I'm unable 
> to find any resource that is being starved.  I turn to `the mighty 
> list` for guidance.
>
> This particular machine is running FreeBSD 6.2.  It has a bce network 
> card that currently connected to a GigE switch with a 10G uplink.  
> Even with all this available bandwidth, this client's application 
> *currently* is not doing any more than 50Mb/sec.  ::  Rule out network 
> congestion.
>
> Moving along to the system ram :: 2 Gigs are in the machine, with very 
> little usage:
> Mem: 724M Active, 874M Inact, 338M Wired, 64M Cache, 112M Buf, 4944K Free
>
> A dual core cpu is in the machine, again with very little usage:
> CPU states:  8.6% user,  0.0% nice,  3.9% system,  2.8% interrupt, 
> 84.7% idle
>
> Disk throughput is negligable at this time (50KB/sec :: 3tps).
>
> I did start off mentioning this machine uses tomcat...so let me 
> continue.  We've setup the java process to use libthr via libmap.conf:
> #############
> cat /etc/libmap.conf
> [java]
> libpthread.so.2 libthr.so.2
> libpthread.so libthr.so
> #############
>
> And added the following to rc.conf::
> ###############
> tomcat55_java_opts="-XX:MaxPermSize=512m -Xmx512M"
> ###############
>
> Apache Configuration:
> ###################
> MaxClients         1024
> ###################
>
> Tomcat Configuration:
> ###################
> maxThreads="1200"
> ###################
>
> ----------------------------------------------------------------
> So onto my question...the slowness being encountered.  My initial 
> thought was the slowness was due to quite possibly the cpu being 
> hammered by numerous requests.  After checking however that's not the 
> case, then I looked at the networking equipment with my networking 
> team, that's ok too.  (I think) That leaves a) thread contention 
> somewhere  b) apache misconfiguration and c) tomcat configuration.
>
> Both myself and 2 other admins have looked over the apache 
> configuration and tomcat configuration and we believe that side of 
> things is probably ok.  That's leaves weird contention in the kernel 
> or userland mutexes or something along those lines.
>
> Here is our current connection count on the external interface ::
> ################################
> sockstat -4cp 80 | wc -l
>     997
> ################################
>
> Here is our connection count on localhost (from apache to tomcat 
> process and vica versa) ::
> ################################
> sockstat -4cp 8009  | wc -l
>    1679
> ################################
> ----------------------------------------------------------------
>
> And finally.....top output:
>
> last pid: 57747;  load averages:  0.79,  1.12,  
> 0.87                                                                                
> up 153+17:56:53 18:36:57
> <#--snip -->
>  PID USERNAME   PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
> 57460 www         96    0  1385M   432M ucond  0   2:44  2.88% java
> 57460 www         96    0  1385M   432M ucond  0   2:44  2.73% java
> 57460 www         96    0  1385M   432M select 0   2:44  0.15% java
> 57460 www         96    0  1385M   432M select 1   2:44  0.15% java
> <#--snip-->
>
> The above lines repeat up to the number of threads we have.  `states` 
> are in ucond and select.
> ----------------------------------------------------------------
>
> Hopefully I hven't bored anyone....yet, and would appreciate any 
> guildance.  Maybe good tools for debugged kernel threads, or a simple 
> slap in the face will do nicely.
>
> Thanks in advance,
> Paul
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to 
> "freebsd-questions-unsubscribe at freebsd.org"


List,

Apologies, but I sever my head in shame as I have found the problem.   
In one word:  KeepAlives.

Thanks!