apache 100% cpu usage

Sat Feb 19 06:34:31 UTC 2011

> -----Original Message-----
> From: Jeremy Chadwick [mailto:freebsd at jdc.parodius.com]
> Sent: Saturday, February 19, 2011 9:01 AM
> To: Payam Poursaied
> Cc: freebsd-apache at freebsd.org
> Subject: Re: apache 100% cpu usage
> 
> On Sat, Feb 19, 2011 at 07:53:31AM +0330, Payam Poursaied wrote:
> > Hi all
> > I faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE.
> > The problem is that httpd processes frequently reach 100% CPU usage and stay
> > at this level of CPU usage.
> > This is a sample of top output.
> >
> >   PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
> > 77051 www          1 118    0   208M 86080K CPU4    4  87.6H 100.00% httpd
> >  9403 www          1 118    0   215M 93340K CPU2    2  65.0H 100.00% httpd
> > 78430 www          1 118    0   226M   100M CPU0    0  58:57 100.00% httpd
> > 95332 www          1  50    0   205M 83624K sbwait 12   0:02  4.49% httpd
> > 89664 www          1  46    0   210M 88144K sbwait 11   0:05  2.59% httpd
> >
> > I think there might be a bug that caused something like endless loop but I
> > am newbie to gdb and debugging and I could not find any relevant point to
> > get into it.
> > I tried to compile apache in debug mode and then wait to see a suspicious
> > process and then attached GDB66 to that cpu intensive httpd process to find
> > out what is going on. It seems the problem is related to something in
> > /lib/libc.so.7 which I could not find out how to compile it in debug mode.
> >
> > The result of "list" and "where" command of GDB are as below, I appreciate
> > if you could point me to right direction to examine this problem deeply.
> > As an extra information, mod_perl is heavily used trough this apache
> >
> > (gdb) l
> > 5640
> > 5641    #ifdef TPF
> > 5642        EBW_AREA input_parms;
> > 5643        ecbptr()->ebrout = PRIMECRAS;
> > 5644        input_parms = * (EBW_AREA *)(&(ecbptr()->ebw000));
> > 5645    #endif
> > 5646
> > 5647        MONCONTROL(0);
> > 5648
> > 5649        common_init();
> > (gdb) w
> > Ambiguous command "w": watch, wh, whatis, where, while, while-stepping,
> > winheight, ws.
> > (gdb) where
> > #0  0x000000080093a9f5 in ?? () from /lib/libc.so.7
> > #1  0x000000080093d505 in ?? () from /lib/libc.so.7
> > #2  0x000000080093d702 in ?? () from /lib/libc.so.7
> > #3  0x000000080094352e in ?? () from /lib/libc.so.7
> > #4  0x0000000800944924 in realloc () from /lib/libc.so.7
> > #5  0x000000080307a50f in Perl_safesysrealloc () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #6  0x00000008030a6156 in Perl_sv_grow () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #7  0x000000080309e32e in Perl_sv_vcatpvfn () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #8  0x00000008030a7b42 in Perl_vnewSVpvf () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #9  0x00000008030a7bf8 in Perl_newSVpvf () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #10 0x000000080303da32 in Perl_gv_fetchmethod_autoload () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #11 0x000000080308edca in S_method_common () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #12 0x000000080308f462 in Perl_pp_method () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #13 0x000000080308eb9e in Perl_runops_standard () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #14 0x0000000803036191 in Perl_call_sv () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #15 0x0000000802eabe36 in perl_call_handler () from
> > /usr/local/libexec/apache/libperl.so
> > #16 0x0000000802eac87a in perl_run_stacked_handlers () from
> > /usr/local/libexec/apache/libperl.so
> > #17 0x0000000802eae2b0 in perl_handler () from
> > /usr/local/libexec/apache/libperl.so
> > #18 0x000000000040ee42 in ap_invoke_handler (r=0x804624060) at
> > http_config.c:476
> > #19 0x0000000000427849 in process_request_internal (r=0x804624060) at
> > http_request.c:1299
> > #20 0x00000000004278a4 in ap_process_request (r=0x804624060) at
> > http_request.c:1315
> > #21 0x000000000041d423 in child_main (child_num_arg=14) at http_main.c:4885
> > #22 0x000000000041d734 in make_child (s=0x800c05060, slot=14,
> > now=1297853738) at http_main.c:5055
> > #23 0x000000000041db6b in perform_idle_server_maintenance () at
> > http_main.c:5256
> > #24 0x000000000041e159 in standalone_main (argc=1, argv=0x7fffffffebd8) at
> > http_main.c:5520
> > #25 0x000000000041e7a4 in main (argc=1, argv=0x7fffffffebd8) at
> > http_main.c:5773
> 
> This looks like you have a mod_perl problem, or a script that's doing
> it.  It's probably not an Apache issue.
> 
> ktrace would be of more help here than gdb.  It will at least show you
> what the children are doing in real-time, syscall-wise.  Correlating
> this with underlying perl code is up to you/your problem, sadly.
> 
> Good luck.
> 

Thank you for your fast response. I used truss and ktrace, but there wasn't any output to show any system calls or ..
may I know how I can get inside and explore the problem? How it is possible to build  /lib/libc.so.7 in debug mode? Or
any general how-to/instruction on debugging is appreciated.

Best Regards
-payam
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5686 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-apache/attachments/20110219/f3ec2396/smime-0001.bin