How to make Apache (2.2.4) less greedy, or Sendmail less polite?
o.greve at axis.nl
Fri May 4 11:26:24 UTC 2007
>It might as well be a search engine spider.
Perhaps it is... By just tailing the httpd-access.log file a few
times over the past 10 minutes or so, at least I already just came
220.127.116.11 - - [04/May/2007:13:12:34 +0200] "GET /olympus/tope/
tope_show_entry.php?event=13&pic=1 HTTP/1.0" 200 3209 "-" "Mozilla/
5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/
18.104.22.168 - - [04/May/2007:13:15:46 +0200] "GET /olympus/tope/
tope_show_entry.php?pic=22&event=7 HTTP/1.1" 200 3792 "-" "Mozilla/
5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
22.214.171.124 - - [04/May/2007:13:15:52 +0200] "GET /clomid HTTP/1.0"
404 345 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
So, there ya go: it looks like Yahoo!, Google AND MSNsearch are
battering my server pretty much all at once (though I don't see too
many traces of either of them, so perhaps they're not all too hard on
Hmmm, perhaps time for a "robots.txt" file.
Then, doing just some more tailing on the access log shows that the
actual load is spread over several sites, with about 4 of the top
sites being called from a variety of browsers from all sorts of IP
addresses. Another assumption then becomes that perhaps as soon as
the USA "wakes up", the sites simply get a lot of real visitors too,
hence causing a (legit) high load...
Still, this is precisely why I'd like to see which scripts are
causing heavy load, such that I can perhaps better tune them.
More information about the freebsd-amd64