How to make Apache (2.2.4) less greedy, or Sendmail less polite?

Olaf Greve o.greve at axis.nl
Thu May 3 22:28:53 UTC 2007


Hi,

Recently I upgraded my Apache 1.3.33 webserver to Apache 2.2.4, and  
ever since, I noticed that it is acting in such a way that it often  
is VERY greedy with my server's resources.
Quite often, when running "top", a list that is as the one that  
appears at the bottom of this e-mail is shown: indeed pretty much  
solely httpd instances, that for extended periods of time almost  
continously pull the CPU to close to 100%, and that also consume a  
lot of the memory resources... Strangely enough, at other times the  
CPU load is just slightly above 0%, say 0.4% or so...

Apart from the fact that it "doesn't feel right" to see the CPU for  
substantial amounts of time, almost constantly close to 100%, there  
is a further issue, being that sendmail rejects connections when the  
server load is (too) high. This is very annoying, as e-mail is also a  
crucial part of the server's functionality, and I don't want sendmail  
to reject connections, each and every time that Apache goes berserk.

Now, the machine in question, is an AMD-64 machine, and it runs the  
AMD-64 version of FreeBSD (5.4-release) with a custom kernel.
Surely, Apache can be reconfigured such that it doesn't behave so  
selfishly, and leaves a decent amount of resources for other stuff  
(such as sendmail) on the machine too.

What I'm basically trying to find out is:
1-Is this normal, or can this perhaps be some (brute force) hack  
attempt, where something is pounding Apache heavily, trying to find/ 
exploit some security risk?
2-How can I inspect exactly what each httpd instance is doing (i.e.  
which request it is serving)?
3-How to best configure Apache 2.2.4 such that it will never use more  
than a specific amount of the system's resources (e.g. a CPU usage  
limit of 75%, and a memory limit of say 1GB)? It would be my guess  
that the amount of "MaxClients" should be lowered, but is that  
sufficient (note: current httpd-mpm.conf settings apper at the end of  
this e-mail, and indicate an amount of 150), and will that not  
somehow (all too) negatively affect the way Apache handles requests?
4-How to perhaps tell sendmail to be a bit more selfish, and stop it  
from rejecting connections for extended periods of time? (note: we  
all know just how much "fun" it can be to configure Sendmail :P so  
for now I've only included (a shortened version of the) RX daemon  
config file, and hope someone can give me a good pointer for this -  
or tell me where else to look).
5-When sendmail rejects (incoming) connections, does mail actually  
get lost, or will it (always) be handled later, when the server is  
less occupied?

Cheers, and tnx in advance!
Olafo


PS: I hope anyone can give me some good ideas, and for completeness  
sake, I've copied some additional information that may give an  
insight into the issues:


1) The Sendmail "rejecting connections" issue:
ps auxww | grep sendmail
root    2259  0.0  0.0  9480   668  ??  Ss   20Apr07   0:38.17  
sendmail: rejecting connections on daemon MSA: load average: 59  
(sendmail)
smmsp   2261  0.0  0.0 13628   760  ??  S    20Apr07   1:40.56  
sendmail: running queue: /var/spool/mqueue-rx (sendmail)
root    2262  0.0  0.0  9480   704  ??  Ss   20Apr07   0:37.85  
sendmail: accepting connections (sendmail)
smmsp   2265  0.0  0.0  9344   608  ??  Is   20Apr07   0:01.33  
sendmail: Queue runner at 00:10:00 for /var/spool/clientmqueue (sendmail)
root   91503  0.0  0.0   428   320  p0  D+    7:23PM   0:00.00 grep  
sendmail

2) "top" output (partial), during (apparent) heavy load:
last pid: 91504;  load averages: 58.76, 59.21,  
60.20                                               up 13+07:02:40   
19:24:50
163 processes: 61 running, 102 sleeping
CPU states: 98.8% user,  0.0% nice,  0.4% system,  0.8% interrupt,   
0.0% idle
Mem: 1299M Active, 204M Inact, 289M Wired, 63M Cache, 214M Buf, 39M Free
Swap: 2021M Total, 922M Used, 1099M Free, 45% Inuse, 128K In

   PID USERNAME        PRI NICE   SIZE    RES STATE    TIME   WCPU     
CPU COMMAND
91459 www             124    0   141M 15136K RUN      0:02  5.52%   
5.52% httpd
91352 www             119    0   139M 12596K select   0:14  3.61%   
3.61% httpd
91455 www             124    0   167M 41960K RUN      0:03  3.61%   
3.61% httpd
91461 www             124    0   141M 15128K RUN      0:03  1.37%   
1.37% httpd
91126 www             124    0   158M 19520K RUN      1:46  0.83%   
0.83% httpd
91139 www             124    0   158M 19532K RUN      1:43  0.83%   
0.83% httpd
91152 www             124    0   195M 19396K RUN      1:40  0.83%   
0.83% httpd
91175 www             124    0   170M 44524K RUN      1:02  0.83%   
0.83% httpd
90387 www             124    0   170M 27548K RUN      5:19  0.78%   
0.78% httpd
90529 www             124    0   195M 24584K RUN      4:49  0.78%   
0.78% httpd
90665 www             124    0   167M 41804K RUN      3:29  0.78%   
0.78% httpd
90897 www             124    0   181M 23964K RUN      2:10  0.78%   
0.78% httpd
91069 www             124    0   158M 20128K RUN      1:53  0.78%   
0.78% httpd
91100 www             124    0   162M 21284K RUN      1:49  0.78%   
0.78% httpd
91125 www             124    0   162M 22976K RUN      1:46  0.78%   
0.78% httpd
91092 www             124    0   162M 23972K RUN      1:45  0.78%   
0.78% httpd
91133 www             124    0   162M 17928K RUN      1:44  0.78%   
0.78% httpd
91135 www             124    0   158M 19668K RUN      1:43  0.78%   
0.78% httpd
89973 www             124    0   158M 18212K RUN      8:15  0.73%   
0.73% httpd
90269 www             124    0   158M 19940K RUN      6:40  0.73%   
0.73% httpd
90254 www             124    0   162M 25584K RUN      5:58  0.73%   
0.73% httpd
90464 www             124    0   195M 27464K RUN      4:51  0.73%   
0.73% httpd
91027 www             124    0   181M 24104K RUN      2:09  0.73%   
0.73% httpd
91072 www             124    0   158M 17456K RUN      1:53  0.73%   
0.73% httpd
91098 www             124    0   158M 19780K RUN      1:48  0.73%   
0.73% httpd
91119 www             124    0   162M 23436K RUN      1:45  0.73%   
0.73% httpd
91121 www             124    0   162M 23088K RUN      1:45  0.73%   
0.73% httpd
91130 www             124    0   158M 19880K RUN      1:44  0.73%   
0.73% httpd
91145 www             124    0   158M 19788K RUN      1:42  0.73%   
0.73% httpd
91141 www             124    0   181M 23852K RUN      1:40  0.73%   
0.73% httpd
91131 www             124    0   167M 41904K RUN      0:37  0.73%   
0.73% httpd
91312 www             124    0   157M 31908K RUN      0:37  0.73%   
0.73% httpd
89962 www             124    0   179M 35048K RUN     10:58  0.68%   
0.68% httpd
89964 www             124    0   179M 39184K RUN      9:51  0.68%   
0.68% httpd

3) "apachectl -l" output:
Compiled in modules:
   core.c
   prefork.c
   http_core.c
   mod_so.c

4) Apache's usr/local/etc/apache22/extra/httpd-mpm.conf settings  
regarding the amount of clients etc. (note: I assume the "prefork  
MPM" to be the one that is used, so I removed the BeOS, NetWare and  
OS2 MPM configuration stuff, to save space):
#
# Server-Pool Management (MPM specific)
#

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
#
# Note that this is the default PidFile for most MPMs.
#
<IfModule !mpm_netware_module>
     PidFile /var/run/httpd.pid
</IfModule>

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
<IfModule !mpm_winnt_module>
<IfModule !mpm_netware_module>
LockFile /var/log/accept.lock
</IfModule>
</IfModule>

#
# Only one of the below sections will be relevant on your
# installed httpd.  Use "apachectl -l" to find out the
# active mpm.
#

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept  
spare
# MaxSpareServers: maximum number of server processes which are kept  
spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process  
serves
<IfModule mpm_prefork_module>
     StartServers          5
     MinSpareServers       5
     MaxSpareServers      10
     MaxClients          150
     MaxRequestsPerChild   0
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server  
process
# MaxRequestsPerChild: maximum number of requests a server process  
serves
<IfModule mpm_worker_module>
     StartServers          2
     MaxClients          150
     MinSpareThreads      25
     MaxSpareThreads      75
     ThreadsPerChild      25
     MaxRequestsPerChild   0
</IfModule>

# [cut BeOS MPM, NetWare MPM, and OS/2 MPM configuration, as these  
should not be in use]


5) /etc/mail/<servername>-rx.mc:
millennics# more ./millennics.nl-rx.mc
dnl To be used for MTA-RX, the first MTA instance (receiving mail)

dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN  
calls.

divert(-1)

#
#  [cut the general text]
#

divert(0)
VERSIONID(`$FreeBSD: src/etc/sendmail/freebsd.mc,v 1.28 2003/04/18  
01:25:41 gshapiro Exp $')
OSTYPE(freebsd5)
DOMAIN(generic)
dnl Specify here also access controls, relayable domains, anti-spam  
measures
dnl including milter settings if needed, mail submission settings,  
client
dnl authentication, resource controls, maximum mail size and header  
size,
dnl confMIN_FREE_BLOCKS, and other settings needed for receiving mail.
dnl
dnl NOTE:
dnl   confMIN_FREE_BLOCKS at MTA-RX should be kept higher than the same
dnl   setting at MTA-TX, to quench down clients when disk space is low,
dnl   and not to stop processing the already received mail.
dnl
dnl In particular, here are some settings to be considered:
dnl   ( see also http://www.sendmail.org/m4/anti_spam.html )
dnl
dnl FEATURE(`access_db',`hash -T<TMPF> /etc/mail/access.db')
dnl VIRTUSER_DOMAIN(`sub1.example.com')dnl  list valid users here
dnl VIRTUSER_DOMAIN(`sub2.example.com')dnl  list valid users here
dnl FEATURE(`virtusertable', `hash /etc/mail/virtusertable')
dnl define(`confUSERDB_SPEC', `/etc/mail/userdb.db')
dnl FEATURE(`blacklist_recipients')
dnl INPUT_MAIL_FILTER(...)
dnl define(`confPRIVACY_FLAGS', `noexpn,novrfy,authwarnings')   
nobodyreturn ?
dnl define(`confDONT_PROBE_INTERFACES')
dnl undefine(`USE_CW_FILE')dnl  cancel use_cw_file feature, no class  
{w} extras
dnl MASQUERADE_AS(...) FEATURE(`allmasquerade') FEATURE 
(`masquerade_envelope')
dnl define(`confTO_IDENT', `0')dnl  Disable IDENT
dnl define(`confMAX_MESSAGE_SIZE',`10485760')
dnl define(`confMAX_MIME_HEADER_LENGTH', `256/128')
dnl define(`confNO_RCPT_ACTION', `add-to-undisclosed')
dnl FEATURE(`nocanonify', ...)
dnl define(`confBIND_OPTS', ...)
dnl define(`confTO_RESOLVER_*... )
dnl define(`confDELAY_LA,    8)
dnl define(`confREFUSE_LA', 12)
dnl define(`confMIN_FREE_BLOCKS', `10000')
dnl define(`confDEF_USER_ID', ...)

FEATURE(access_db, `hash -o -T<TMPF> /etc/mail/access')
FEATURE(blacklist_recipients)
FEATURE(local_lmtp)
FEATURE(mailertable, `hash -o /etc/mail/mailertable')
FEATURE(virtusertable, `hash -o /etc/mail/virtusertable')

dnl [cut relaying, and black(hole) lists, etc., as these are all  
commented out]

dnl Uncomment the first line to change the location of the default
dnl /etc/mail/local-host-names and comment out the second line.
dnl define(`confCW_FILE', `-o /etc/mail/sendmail.cw')
define(`confCW_FILE', `-o /etc/mail/local-host-names')

dnl Uncomment both of the following lines to listen on IPv6 as well  
as IPv4
dnl DAEMON_OPTIONS(`Name=IPv4, Family=inet')
dnl DAEMON_OPTIONS(`Name=IPv6, Family=inet6')

define(`confBIND_OPTS', `WorkAroundBrokenAAAA')
define(`confNO_RCPT_ACTION', `add-to-undisclosed')
define(`confPRIVACY_FLAGS', `authwarnings,noexpn,novrfy')

define(`confRUN_AS_USER',`smmsp:smmsp')dnl  Drop privileges (see  
SECURITY NOTE)

define(`confPID_FILE', `/var/run/sendmail-rx.pid')dnl  Non-default  
pid file
define(`STATUS_FILE', `/etc/mail/stat-rx')dnl    Non-default stat file
define(`QUEUE_DIR', `/var/spool/mqueue-rx')dnl   Non-default queue area
define(`confQUEUE_SORT_ORDER',`Modification')dnl Modif or Random are  
reasonable

dnl Match the number of queue runners (R=) to the number of amavisd- 
new child
dnl processes ($max_servers). 2 to 7 OK, 10 is plenty, 20 is too many
QUEUE_GROUP(`mqueue', `P=/var/spool/mqueue-rx, R=2, F=f')dnl

dnl Direct all mail to be forwarded to amavisd-new at 127.0.0.1:10024
FEATURE(stickyhost)dnl  Keep envelope addr "u at local.host" when fwd to  
MAIL_HUB
define(`MAIL_HUB',  `esmtp:[127.0.0.1]')dnl  Forward all local mail  
to amavisd
define(`SMART_HOST',`esmtp:[127.0.0.1]')dnl  Forward all other mail  
to amavisd

define(`confDELIVERY_MODE',`q')dnl     Delivery mode: queue only (a  
must,
dnl  ... otherwise the advantage of this setup of being able to specify
dnl  ... the number of queue runners is lost)
define(`ESMTP_MAILER_ARGS',`TCP $h 10024')dnl  To tcp port 10024  
instead of 25
MODIFY_MAILER_FLAGS(`ESMTP', `+z')dnl  Speak LMTP (this is optional)
define(`SMTP_MAILER_MAXMSGS',`10')dnl  Max no. of msgs in a single  
connection
define(`confTO_DATAFINAL',`20m')dnl    20 minute timeout for content  
checking
DAEMON_OPTIONS(`Name=MTA-RX')dnl       Daemon name used in logged  
messages

dnl Disable local delivery, as all local mail will go to MAIL_HUB
undefine(`ALIAS_FILE')dnl     No aliases file, all local mail goes to  
MAIL_HUB
define(`confFORWARD_PATH')dnl Empty search path for .forward files
undefine(`UUCP_RELAY')dnl
undefine(`BITNET_RELAY')dnl
undefine(`DECNET_RELAY')dnl

MAILER(smtp)






More information about the freebsd-amd64 mailing list