shared mem and panics when out of PV Entries

[Andrew Kinney]

On 25 Mar 2003, at 17:56, Igor Sysoev wrote:

> > So, what's the best approach to limiting memory shared via fork() or
> > reducing PV Entry usage by that memory?  Is there something I can do
> > with the kernel config or sysctl to accomplish this?
> 
> No, as far as I know there's no way to do it.
> The irony is that you do not need the most of these PV entries because
> you are not swaping.
> 

My thoughts exactly.  I suppose not all that many people run well  
used web servers with 4GB of RAM, so there wouldn't be any 
reason for this issue to come up on a regular basis.  

I'm going to expose my newbness here with respect to BSD 
memory management, but could the number of files served and 
filesystem caching have something to do with the PV Entry usage 
by Apache?  We've got around 1.2 million files served by this 
Apache.  Could it be that the extensive PV Entry usage has 
something to do with that?  Obviously, not all are accessed all the 
time, but it wouldn't take a very large percentage of them being 
accessed to cause issues if filesystem caching is in any way 
related to PV Entry usage by Apache.

I remember reading somewhere (sorry, didn't keep track of the link) 
that someone running a heavily used Squid proxy had a very 
similar issue with running out of PV Entries as they got more and 
more files in the cache.  Squid is basically a modified Apache with 
proxy caching turned on.

> I think you should try to decrease memory that shared between Apache
> processes. If you can not change scripts then the single method is to
> decrease number of Apache processes while keeping to handle the
> current workload: 1) disable keepalive if it enabled; 2) set Apache
> behind reverse-proxy server that frees Apache processes
>    as soon as proxy get the whole responses.

We had keepalive set to the default of "on" (at least default for this 
install) with the default keepalive timeout of 15 seconds.

Dropping the keepalive timeout down to 3 seconds has 
dramatically reduced the number of Apache processes required to 
serve the load.  With the new settings, we're averaging 30 to 80 
Apache processes, which is much more manageable in terms of 
memory usage, though we weren't anywhere near running out of 
physical RAM prior to this.  We're servicing a little over 1000 
requests per minute, which by some standards isn't a huge amount.

We're still seeing quite heavy PV Entry usage, though.  The 
reduced number of Apache processes (by more than half) doesn't 
seem to have appreciably reduced PV Entry usage versus the 
previous settings, so I suspect I may have been wrong about 
memory sharing as the culprit for the PV Entry usage.  This 
observation may just be coincidence, but the average PV Entry 
usage seems to have gone up by a couple million entries since the 
changes to the Apache config.

Time will tell if the PV Entries are still getting hit hard enough to 
cause panics due to running out of them.  They're supposed to get 
forcibly recycled at 90% utilization from what I see in the kernel 
code, so if we never get above 90% utilization I guess I could 
consider the issue resolved.

What other things in Apache (besides memory sharing via PHP 
and/or mod_perl) could generate PV Entry usage on a massive 
scale?

Sincerely,
Andrew Kinney
President and
Chief Technology Officer
Advantagecom Networks, Inc.
http://www.advantagecom.net