swap getting consumed

[Mark Edwards]

On Jan 23, 2005, at 7:27 PM, J65nko BSD wrote:

> On Sun, 23 Jan 2005 17:33:35 -0800, Mark Edwards 
> <mark at antsclimbtree.com> wrote:
>> On Jan 23, 2005, at 3:48 PM, J65nko BSD wrote:
>>
>>> On Sun, 23 Jan 2005 14:53:38 -0800, Mark Edwards
>>> <mark at antsclimbtree.com> wrote:
>>>> I have posted about this problem a couple of times with not much
>>>> response, I'm afraid, but here is a different take on it perhaps.  I
>>>> have a 4.10p5 running and for roughly the last two months my swap
>>>> space
>>>> has been getting eaten uncontrollably.  The only clue I have is that
>>>> it
>>>> resets when I restart Apache (1.3.33), and so if I set up a cron job
>>>> to
>>>> restart Apache every day or hour the problem is contained:
>>>>
>>>> https://secure.antsclimbtree.com/mrtg/ants.swap-year.png
>>>>
>>>> Oddly, my physical memory doesn't seem to be generally affected:
>>>>
>>>> https://secure.antsclimbtree.com/mrtg/ants.ram-year.png
>>>>
>>>> I have 128MB of physical RAM and 384MB of swap.
>>>>
>>>> The only clue I have with Apache is a lot of this in
>>>> /var/log/messages:
>>>>
>>>> Jan 21 18:25:06 lilbuddy /kernel: pid 68446 (httpd), uid 80: exited 
>>>> on
>>>> signal 6
>>>>
>>>> But I'm not sure what to make of it.  The only thing Google turns up
>>>> related to those messages are notes about CodeRed virus attacks, but
>>>> Apache 1.3.33 is suppose to address that issue.
>>>>
>>>> Does anyone have any suggestion of how to attack this problem?  I'm
>>>> totally stumped.
>>>>
>>>
>>> You don't give much background info about the server. Which modules
>>> Apache is using, mod_perl, mod_ssl?, which applications it runs,
>>> WebMail?
>>>
>>> Cannot you see in the Apache logs, which pages are being served 
>>> before
>>> the time of the signal 6 messages?
>>> Google for "Apache+memory+leak" and include every module or app you
>>> are running..
>>>
>>> Gather more info what the system is doing exactly. Write a script for
>>> cron to run "netstat -m", "sockstat -4", "ps -ax". at regular times
>>> and have the output mailed to you.
>>>
>>> BTW 128 MB memory for a webserver is not much these days ;)
>>>
>>> =Adriaan=
>>
>> Thanks for the reply.  I am running apache+mod_ssl-1.3.33+2.8.22, with
>> mod_php4 and mod_dav.
>>
>> As far as web apps, I run Squirrelmail, phpbb2, MT-2.661, Mailman,
>> analog, webalizer.  The weird thing is that I ran all of those except
>> phpbb2 for years with no problem.  Perhaps it is phpbb2's doing, but I
>> can't find a correlation between that and the signal 6 messages, and 
>> in
>> any case phpbb2 hardly gets used.  I can watch swap go up and up
>> without anyone actually using phpbb2.
>>
>> Doing Google on Apache+memory+leak as you suggest leads to people
>> describing similar situations, where Apache eats more and more swap 
>> and
>> has to be restarted, but I can't find anyone posing a solution to the
>> problem.
>>
>> I will just keep monitoring it I guess, looking for clues.  128MB 
>> might
>> not be much, but it has worked fine for running essentially the same
>> set of software for years, and all of a sudden there's a problem.  I
>> wish I could tie it to some specific update or change, but I can't.
>> The closest I can come is turning on softupdates and installing 
>> phpbb2,
>> and neither of those seems to correlate.
>>
>> Very odd...
>
> Have you seen this security warning for phpbb?
> http://www.auscert.org.au/render.html?it=4653

Just want to close this thread by saying that I seem to have solved the 
problem, although I'm not 100% sure what did it.  I'm pretty sure it 
was a recompile of mm-1.3.1.  I tried this because I noticed that it 
was the only required component for apache that was last updated in 
mid-November, right around when this nonsense started.  All the other 
required ports had been updated more recently, so I figured that 
perhaps something was amiss when that one got compiled in November.  
Plus, the fact that mm deals with memory management.

That appears to have been the culprit.  I'm running along with swap 
moving between the 100MB and 200MB range, with no big spikes, and RAM 
at a steady 64MB out of 128MB.  Performance is steady and back to 
normal.  Hurrah!

A 180Mhz Pentium Pro with 128MB RAM is no speed demon, mind you, but it 
serves web pages, hosts email, and routes pretty damn fine for a 
machine that was literally in the garbage when I rescued it.  PHP is a 
bit slow, but still totally acceptable, and faster than the 256Kbps 
outgoing connection.

I also doubled my swap from 400MB on one drive to 800MB between two 
drives.  And I stripped down my kernel a bit, removing lots of options 
that aren't used on this box.  Those might have affected things as 
well, but my hunch is it was mm-1.3.1 that was out of whack.

Anyway, all's well that ends well.  Thanks!