overloaded webserver: nfs wait issue?

[Norberto Meijome]

N.J. Thomas wrote:
> We have a website with moderately high traffic, load balanced among 3
> webservers.
> 
> During peak traffic times however (when the volume is higher than
> normal), the load shoots up to over a 100, and the site crawls to its
> knees.
> 
> We set up a script to take snapshots of top every 20 seconds. Here is
> what it looks like when everthing is normal:
> 
>          127
>     last pid: 12003;  load averages:  0.93,  1.36,  1.35  up 41+04:22:14    14:00:23
>     243 processes: 12 running, 230 sleeping, 1 zombie
> 
>     Mem: 222M Active, 74M Inact, 186M Wired, 16M Cache, 111M Buf, 503M Free
>     Swap: 2048M Total, 16M Used, 2032M Free
> 
> 
[...]

> When the load shoots up, the number of http clients hits Apache's
> MaxClients setting, here is what top shows:
> 
>     last pid: 12407;  load averages: 87.84, 51.91, 27.52  up 41+04:40:51    14:19:00
>     268 processes: 2 running, 266 sleeping
> 
>     Mem: 715M Active, 68M Inact, 187M Wired, 29M Cache, 111M Buf, 2100K Free
>     Swap: 2048M Total, 272M Used, 1776M Free, 13% Inuse
> 

[...]

> 
> The state of all the httpd prcesses are "nfsrcv". Does this mean the
> bottleneck is at the NFS server that hosts the htdocs (and PHP scripts)
> or just that the server is low on memory?

Hi Thomas,

What's your MaxClients set to?

Please define your values for "lot of traffic".

What CPU? RAM (512MB seems a bit low nowadays)? Disks?

I dont think i can give much advice on the NFS side of things but in the 
meantime I would :
  - increase # of MaxClients (the default is RIDICULOUSLY small, 
specially in 1.3. You will probably have to recompile with a new max.

  - Look at what the PHP scripts do : i.e., is there anything under your 
control that can be improved?

  - Install a PHP accelerator (like IonCube's, or Zend's). Or some 
reverse proxies if you feel like playing with the whole design of your 
architecture.

  - if you use mod_gzip, make sure you dont' compress on the fly - it 
takes a huge hit on the CPU. (Hint: You can pre-compress the files )

  - You RAM seems OK ... you may want to tweak some sysctl or memory 
settings in Apache (I seem to remember in 1.3 some to do with MMap,  but 
i could be wrong) ... or just add more RAM. Check vmstat (or systat -vm 
1) to see how much swapping is going on.

  - Make sure you have tweaked your network sysctl settings (like send + 
receive buffers, depending on the kind of traffic you get), reduce your 
close timeouts,etc.

  - make use of the kernel modules accf_http.ko and accf_data.ko.

  - disable from Apache ANYTHING that is NOT needed. Enable stuff only 
as  needed on a per virtual host basis.

Hope this helps.
Beto