Load Balancing

[Chuck Swiger]

Michael Jeung wrote:
> We currently use DNS round-robin to balance traffic to servers.  We've 
> recently run into situations where multiple search engine spiders are 
> crawling our webservers.  They appear to be targeting specific 
> webservers by IP address.  This defeats DNS round-robin and as a 
> result 1 of our webservers starts responding very slowly since it's 
> full of search engine spiders.

Set up a /robots.txt file in your webserver root which contains 
appropriate Disallow or Crawl-delay entries to reduce the rate at which 
spiders make requests, or eliminate such traffic entirely.  For example:

http://www.pkix.net/robots.txt

-- 
-Chuck