How to make Apache (2.2.4) less greedy, or Sendmail less polite?
Olaf Greve
o.greve at axis.nl
Thu May 3 22:15:36 UTC 2007
Hi,
Recently I upgraded my Apache 1.3.33 webserver to Apache 2.2.4, and
ever since, I noticed that it is acting in such a way that it often
is VERY greedy with my server's resources.
Quite often, when running "top", a list that is as the one that
appears at the bottom of this e-mail is shown: indeed pretty much
solely httpd instances, that for extended periods of time almost
continously pull the CPU to close to 100%, and that also consume a
lot of the memory resources... Strangely enough, at other times the
CPU load is just slightly above 0%, say 0.4% or so...
Apart from the fact that it "doesn't feel right" to see the CPU for
substantial amounts of time, almost constantly close to 100%, there
is a further issue, being that sendmail rejects connections when the
server load is (too) high. This is very annoying, as e-mail is also a
crucial part of the server's functionality, and I don't want sendmail
to reject connections, each and every time that Apache goes berserk.
Now, the machine in question, is an AMD-64 machine, and it runs the
AMD-64 version of FreeBSD (5.4-release) with a custom kernel.
Surely, Apache can be reconfigured such that it doesn't behave so
selfishly, and leaves a decent amount of resources for other stuff
(such as sendmail) on the machine too.
What I'm basically trying to find out is:
1-Is this normal, or can this perhaps be some (brute force) hack
attempt, where something is pounding Apache heavily, trying to find/
exploit some security risk?
2-How can I inspect exactly what each httpd instance is doing (i.e.
which request it is serving)?
3-How to best configure Apache 2.2.4 such that it will never use more
than a specific amount of the system's resources (e.g. a CPU usage
limit of 75%, and a memory limit of say 1GB)? It would be my guess
that the amount of "MaxClients" should be lowered, but is that
sufficient (note: current httpd-mpm.conf settings apper at the end of
this e-mail, and indicate an amount of 150), and will that not
somehow (all too) negatively affect the way Apache handles requests?
4-How to perhaps tell sendmail to be a bit more selfish, and stop it
from rejecting connections for extended periods of time? (note: we
all know just how much "fun" it can be to configure Sendmail :P so
for now I've only included (a shortened version of the) RX daemon
config file, and hope someone can give me a good pointer for this -
or tell me where else to look).
5-When sendmail rejects (incoming) connections, does mail actually
get lost, or will it (always) be handled later, when the server is
less occupied?
Cheers, and tnx in advance!
Olafo
PS: I hope anyone can give me some good ideas, and for completeness
sake, I've copied some additional information that may give an
insight into the issues:
1) The Sendmail "rejecting connections" issue:
ps auxww | grep sendmail
root 2259 0.0 0.0 9480 668 ?? Ss 20Apr07 0:38.17
sendmail: rejecting connections on daemon MSA: load average: 59
(sendmail)
smmsp 2261 0.0 0.0 13628 760 ?? S 20Apr07 1:40.56
sendmail: running queue: /var/spool/mqueue-rx (sendmail)
root 2262 0.0 0.0 9480 704 ?? Ss 20Apr07 0:37.85
sendmail: accepting connections (sendmail)
smmsp 2265 0.0 0.0 9344 608 ?? Is 20Apr07 0:01.33
sendmail: Queue runner at 00:10:00 for /var/spool/clientmqueue (sendmail)
root 91503 0.0 0.0 428 320 p0 D+ 7:23PM 0:00.00 grep
sendmail
2) "top" output (partial), during (apparent) heavy load:
last pid: 91504; load averages: 58.76, 59.21,
60.20 up 13+07:02:40
19:24:50
163 processes: 61 running, 102 sleeping
CPU states: 98.8% user, 0.0% nice, 0.4% system, 0.8% interrupt,
0.0% idle
Mem: 1299M Active, 204M Inact, 289M Wired, 63M Cache, 214M Buf, 39M Free
Swap: 2021M Total, 922M Used, 1099M Free, 45% Inuse, 128K In
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU
CPU COMMAND
91459 www 124 0 141M 15136K RUN 0:02 5.52%
5.52% httpd
91352 www 119 0 139M 12596K select 0:14 3.61%
3.61% httpd
91455 www 124 0 167M 41960K RUN 0:03 3.61%
3.61% httpd
91461 www 124 0 141M 15128K RUN 0:03 1.37%
1.37% httpd
91126 www 124 0 158M 19520K RUN 1:46 0.83%
0.83% httpd
91139 www 124 0 158M 19532K RUN 1:43 0.83%
0.83% httpd
91152 www 124 0 195M 19396K RUN 1:40 0.83%
0.83% httpd
91175 www 124 0 170M 44524K RUN 1:02 0.83%
0.83% httpd
90387 www 124 0 170M 27548K RUN 5:19 0.78%
0.78% httpd
90529 www 124 0 195M 24584K RUN 4:49 0.78%
0.78% httpd
90665 www 124 0 167M 41804K RUN 3:29 0.78%
0.78% httpd
90897 www 124 0 181M 23964K RUN 2:10 0.78%
0.78% httpd
91069 www 124 0 158M 20128K RUN 1:53 0.78%
0.78% httpd
91100 www 124 0 162M 21284K RUN 1:49 0.78%
0.78% httpd
91125 www 124 0 162M 22976K RUN 1:46 0.78%
0.78% httpd
91092 www 124 0 162M 23972K RUN 1:45 0.78%
0.78% httpd
91133 www 124 0 162M 17928K RUN 1:44 0.78%
0.78% httpd
91135 www 124 0 158M 19668K RUN 1:43 0.78%
0.78% httpd
89973 www 124 0 158M 18212K RUN 8:15 0.73%
0.73% httpd
90269 www 124 0 158M 19940K RUN 6:40 0.73%
0.73% httpd
90254 www 124 0 162M 25584K RUN 5:58 0.73%
0.73% httpd
90464 www 124 0 195M 27464K RUN 4:51 0.73%
0.73% httpd
91027 www 124 0 181M 24104K RUN 2:09 0.73%
0.73% httpd
91072 www 124 0 158M 17456K RUN 1:53 0.73%
0.73% httpd
91098 www 124 0 158M 19780K RUN 1:48 0.73%
0.73% httpd
91119 www 124 0 162M 23436K RUN 1:45 0.73%
0.73% httpd
91121 www 124 0 162M 23088K RUN 1:45 0.73%
0.73% httpd
91130 www 124 0 158M 19880K RUN 1:44 0.73%
0.73% httpd
91145 www 124 0 158M 19788K RUN 1:42 0.73%
0.73% httpd
91141 www 124 0 181M 23852K RUN 1:40 0.73%
0.73% httpd
91131 www 124 0 167M 41904K RUN 0:37 0.73%
0.73% httpd
91312 www 124 0 157M 31908K RUN 0:37 0.73%
0.73% httpd
89962 www 124 0 179M 35048K RUN 10:58 0.68%
0.68% httpd
89964 www 124 0 179M 39184K RUN 9:51 0.68%
0.68% httpd
3) "apachectl -l" output:
Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c
4) Apache's usr/local/etc/apache22/extra/httpd-mpm.conf settings
regarding the amount of clients etc. (note: I assume the "prefork
MPM" to be the one that is used, so I removed the BeOS, NetWare and
OS2 MPM configuration stuff, to save space):
#
# Server-Pool Management (MPM specific)
#
#
# PidFile: The file in which the server should record its process
# identification number when it starts.
#
# Note that this is the default PidFile for most MPMs.
#
<IfModule !mpm_netware_module>
PidFile /var/run/httpd.pid
</IfModule>
#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
<IfModule !mpm_winnt_module>
<IfModule !mpm_netware_module>
LockFile /var/log/accept.lock
</IfModule>
</IfModule>
#
# Only one of the below sections will be relevant on your
# installed httpd. Use "apachectl -l" to find out the
# active mpm.
#
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept
spare
# MaxSpareServers: maximum number of server processes which are kept
spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process
serves
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
</IfModule>
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server
process
# MaxRequestsPerChild: maximum number of requests a server process
serves
<IfModule mpm_worker_module>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
# [cut BeOS MPM, NetWare MPM, and OS/2 MPM configuration, as these
should not be in use]
5) /etc/mail/<servername>-rx.mc:
millennics# more ./millennics.nl-rx.mc
dnl To be used for MTA-RX, the first MTA instance (receiving mail)
dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN
calls.
divert(-1)
#
# [cut the general text]
#
divert(0)
VERSIONID(`$FreeBSD: src/etc/sendmail/freebsd.mc,v 1.28 2003/04/18
01:25:41 gshapiro Exp $')
OSTYPE(freebsd5)
DOMAIN(generic)
dnl Specify here also access controls, relayable domains, anti-spam
measures
dnl including milter settings if needed, mail submission settings,
client
dnl authentication, resource controls, maximum mail size and header
size,
dnl confMIN_FREE_BLOCKS, and other settings needed for receiving mail.
dnl
dnl NOTE:
dnl confMIN_FREE_BLOCKS at MTA-RX should be kept higher than the same
dnl setting at MTA-TX, to quench down clients when disk space is low,
dnl and not to stop processing the already received mail.
dnl
dnl In particular, here are some settings to be considered:
dnl ( see also http://www.sendmail.org/m4/anti_spam.html )
dnl
dnl FEATURE(`access_db',`hash -T<TMPF> /etc/mail/access.db')
dnl VIRTUSER_DOMAIN(`sub1.example.com')dnl list valid users here
dnl VIRTUSER_DOMAIN(`sub2.example.com')dnl list valid users here
dnl FEATURE(`virtusertable', `hash /etc/mail/virtusertable')
dnl define(`confUSERDB_SPEC', `/etc/mail/userdb.db')
dnl FEATURE(`blacklist_recipients')
dnl INPUT_MAIL_FILTER(...)
dnl define(`confPRIVACY_FLAGS', `noexpn,novrfy,authwarnings')
nobodyreturn ?
dnl define(`confDONT_PROBE_INTERFACES')
dnl undefine(`USE_CW_FILE')dnl cancel use_cw_file feature, no class
{w} extras
dnl MASQUERADE_AS(...) FEATURE(`allmasquerade') FEATURE
(`masquerade_envelope')
dnl define(`confTO_IDENT', `0')dnl Disable IDENT
dnl define(`confMAX_MESSAGE_SIZE',`10485760')
dnl define(`confMAX_MIME_HEADER_LENGTH', `256/128')
dnl define(`confNO_RCPT_ACTION', `add-to-undisclosed')
dnl FEATURE(`nocanonify', ...)
dnl define(`confBIND_OPTS', ...)
dnl define(`confTO_RESOLVER_*... )
dnl define(`confDELAY_LA, 8)
dnl define(`confREFUSE_LA', 12)
dnl define(`confMIN_FREE_BLOCKS', `10000')
dnl define(`confDEF_USER_ID', ...)
FEATURE(access_db, `hash -o -T<TMPF> /etc/mail/access')
FEATURE(blacklist_recipients)
FEATURE(local_lmtp)
FEATURE(mailertable, `hash -o /etc/mail/mailertable')
FEATURE(virtusertable, `hash -o /etc/mail/virtusertable')
dnl [cut relaying, and black(hole) lists, etc., as these are all
commented out]
dnl Uncomment the first line to change the location of the default
dnl /etc/mail/local-host-names and comment out the second line.
dnl define(`confCW_FILE', `-o /etc/mail/sendmail.cw')
define(`confCW_FILE', `-o /etc/mail/local-host-names')
dnl Uncomment both of the following lines to listen on IPv6 as well
as IPv4
dnl DAEMON_OPTIONS(`Name=IPv4, Family=inet')
dnl DAEMON_OPTIONS(`Name=IPv6, Family=inet6')
define(`confBIND_OPTS', `WorkAroundBrokenAAAA')
define(`confNO_RCPT_ACTION', `add-to-undisclosed')
define(`confPRIVACY_FLAGS', `authwarnings,noexpn,novrfy')
define(`confRUN_AS_USER',`smmsp:smmsp')dnl Drop privileges (see
SECURITY NOTE)
define(`confPID_FILE', `/var/run/sendmail-rx.pid')dnl Non-default
pid file
define(`STATUS_FILE', `/etc/mail/stat-rx')dnl Non-default stat file
define(`QUEUE_DIR', `/var/spool/mqueue-rx')dnl Non-default queue area
define(`confQUEUE_SORT_ORDER',`Modification')dnl Modif or Random are
reasonable
dnl Match the number of queue runners (R=) to the number of amavisd-
new child
dnl processes ($max_servers). 2 to 7 OK, 10 is plenty, 20 is too many
QUEUE_GROUP(`mqueue', `P=/var/spool/mqueue-rx, R=2, F=f')dnl
dnl Direct all mail to be forwarded to amavisd-new at 127.0.0.1:10024
FEATURE(stickyhost)dnl Keep envelope addr "u at local.host" when fwd to
MAIL_HUB
define(`MAIL_HUB', `esmtp:[127.0.0.1]')dnl Forward all local mail
to amavisd
define(`SMART_HOST',`esmtp:[127.0.0.1]')dnl Forward all other mail
to amavisd
define(`confDELIVERY_MODE',`q')dnl Delivery mode: queue only (a
must,
dnl ... otherwise the advantage of this setup of being able to specify
dnl ... the number of queue runners is lost)
define(`ESMTP_MAILER_ARGS',`TCP $h 10024')dnl To tcp port 10024
instead of 25
MODIFY_MAILER_FLAGS(`ESMTP', `+z')dnl Speak LMTP (this is optional)
define(`SMTP_MAILER_MAXMSGS',`10')dnl Max no. of msgs in a single
connection
define(`confTO_DATAFINAL',`20m')dnl 20 minute timeout for content
checking
DAEMON_OPTIONS(`Name=MTA-RX')dnl Daemon name used in logged
messages
dnl Disable local delivery, as all local mail will go to MAIL_HUB
undefine(`ALIAS_FILE')dnl No aliases file, all local mail goes to
MAIL_HUB
define(`confFORWARD_PATH')dnl Empty search path for .forward files
undefine(`UUCP_RELAY')dnl
undefine(`BITNET_RELAY')dnl
undefine(`DECNET_RELAY')dnl
MAILER(smtp)
More information about the freebsd-questions
mailing list