How to make Apache (2.2.4) less greedy, or Sendmail less polite?

Olaf Greve o.greve at
Fri May 4 11:26:24 UTC 2007


 >It might as well be a search engine spider.

Perhaps it is... By just tailing the httpd-access.log file a few  
times over the past 10 minutes or so, at least I already just came  
across: - - [04/May/2007:13:12:34 +0200] "GET /olympus/tope/ 
tope_show_entry.php?event=13&pic=1 HTTP/1.0" 200 3209 "-" "Mozilla/ 
5.0 (compatible; Yahoo! Slurp; 
[...] - - [04/May/2007:13:15:46 +0200] "GET /olympus/tope/ 
tope_show_entry.php?pic=22&event=7 HTTP/1.1" 200 3792 "-" "Mozilla/ 
5.0 (compatible; Googlebot/2.1; +" - - [04/May/2007:13:15:52 +0200] "GET /clomid HTTP/1.0"  
404 345 "-" "msnbot/1.0 (+"

So, there ya go: it looks like Yahoo!, Google AND MSNsearch are  
battering my server pretty much all at once (though I don't see too  
many traces of either of them, so perhaps they're not all too hard on  
the machine)...
Hmmm, perhaps time for a "robots.txt" file.

Then, doing just some more tailing on the access log shows that the  
actual load is spread over several sites, with about 4 of the top  
sites being called from a variety of browsers from all sorts of IP  
addresses. Another assumption then becomes that perhaps as soon as  
the USA "wakes up", the sites simply get a lot of real visitors too,  
hence causing a (legit) high load...

Still, this is precisely why I'd like to see which scripts are  
causing heavy load, such that I can perhaps better tune them.


More information about the freebsd-amd64 mailing list