How to make Apache (2.2.4) less greedy, or Sendmail less polite?

[Olaf Greve]

Hi,

 >It might as well be a search engine spider.


Perhaps it is... By just tailing the httpd-access.log file a few  
times over the past 10 minutes or so, at least I already just came  
across:

74.6.70.45 - - [04/May/2007:13:12:34 +0200] "GET /olympus/tope/ 
tope_show_entry.php?event=13&pic=1 HTTP/1.0" 200 3209 "-" "Mozilla/ 
5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/ 
slurp)"
[...]
66.249.66.176 - - [04/May/2007:13:15:46 +0200] "GET /olympus/tope/ 
tope_show_entry.php?pic=22&event=7 HTTP/1.1" 200 3792 "-" "Mozilla/ 
5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
65.55.210.7 - - [04/May/2007:13:15:52 +0200] "GET /clomid HTTP/1.0"  
404 345 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"

So, there ya go: it looks like Yahoo!, Google AND MSNsearch are  
battering my server pretty much all at once (though I don't see too  
many traces of either of them, so perhaps they're not all too hard on  
the machine)...
Hmmm, perhaps time for a "robots.txt" file.

Then, doing just some more tailing on the access log shows that the  
actual load is spread over several sites, with about 4 of the top  
sites being called from a variety of browsers from all sorts of IP  
addresses. Another assumption then becomes that perhaps as soon as  
the USA "wakes up", the sites simply get a lot of real visitors too,  
hence causing a (legit) high load...

Still, this is precisely why I'd like to see which scripts are  
causing heavy load, such that I can perhaps better tune them.

Cheers,
Olafo