Runaway Apache
Mark Edwards
mark at antsclimbtree.com
Sat Dec 18 13:25:01 PST 2004
In the last week or so, my FreeBSD 4.10p5 server has started locking up
every day or so, to the point where it becomes unusable and must be
rebooted to resume service. I've noticed that when it happens, the
following type of thing appears in /var/log/httpd-error.log
[Sat Dec 18 13:00:18 2004] [error] child process 248 still did not
exit, sending a SIGKILL
[Sat Dec 18 13:00:18 2004] [error] child process 464 still did not
exit, sending a SIGKILL
[Sat Dec 18 13:00:18 2004] [error] child process 465 still did not
exit, sending a SIGKILL
[Sat Dec 18 13:00:18 2004] [error] child process 466 still did not
exit, sending a SIGKILL
[Sat Dec 18 13:00:18 2004] [error] child process 554 still did not
exit, sending a SIGKILL
[Sat Dec 18 13:00:18 2004] [error] child process 2121 still did not
exit, sending a SIGKILL
[Sat Dec 18 13:00:18 2004] [error] child process 2126 still did not
exit, sending a SIGKILL
[Sat Dec 18 13:00:18 2004] [error] child process 2129 still did not
exit, sending a SIGKILL
[Sat Dec 18 13:00:18 2004] [error] child process 2130 still did not
exit, sending a SIGKILL
and on and on and on...
So, apparently, Apache is having a problem and taking down the server.
I eventually also see complaints about user 80 exceeding the
kern.maxfiles limit. That's probably when the server really takes a
dump.
I've been monitoring top periodically to see if I can spot the problem,
and an httpd process was consuming 95% of the cpu just now, and sure
enough the above messages were streaming through the log. I also
notice the following:
httpd in free(): warning: chunk is already free
httpd in free(): warning: chunk is already free
httpd in free(): warning: chunk is already free
httpd in free(): warning: chunk is already free
httpd in free(): warning: chunk is already free
httpd in free(): warning: chunk is already free
httpd in free(): warning: chunk is already free
Now, my problem is I'm not sure how to find the source of this problem
and stop it. A google search on those log entries suggests that it may
be an attempt to exploit the Chunk Handling Vulnerability, but my
Apache is newer than the fix for that.
http://httpd.apache.org/info/security_bulletin_20020617.txt
Anyhow, can anyone give me a suggestion on how to troubleshoot this?
Thanks!
Here is the Apache in question:
Server Version: Apache/1.3.33 (Unix) mod_ssl/2.8.22 OpenSSL/0.9.7e
PHP/4.3.10 DAV/1.0.3
More information about the freebsd-questions
mailing list