High load, lots of free memory and processes in devfs state
Stut
stuttle at gmail.com
Sun Jun 15 17:42:19 UTC 2008
Hi,
I'm having a serious problem with one of my FreeBSD servers. It runs
FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the
hardware and it's all OK. There's approximately 400 Apache processes
running and a 2GB memcached instance.
Up until last weekend this server was working perfectly at a level of
HTTP traffic higher than it's currently getting. Last week our
database server (separate server) died and had to be rebuilt. While
this was being done this server hosted the database. This has now been
completed and the PHP app is pointing back at the dedicated DB server.
Top shows the following...
last pid: 26838; load averages: 10.22, 14.06, 13.55 up
2+00:34:47 18:03:43
619 processes: 1 running, 618 sleeping
CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt,
70.0% idle
Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf,
1747M Free
Swap: 8192M Total, 124K Used, 8192M Free
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
26807 80 1 -4 0 79892K 9996K devfs 0 0:00 2.81% httpd
26797 80 1 -4 0 82376K 12592K devfs 1 0:00 2.19% httpd
26791 80 1 -4 0 82376K 12636K devfs 0 0:00 1.85% httpd
26783 80 1 -4 0 82392K 12640K devfs 3 0:00 1.84% httpd
26780 80 1 -4 0 79892K 10036K devfs 0 0:00 1.51% httpd
26801 80 1 -4 0 81292K 11104K devfs 3 0:00 1.50% httpd
26786 80 1 -4 0 80456K 10796K devfs 1 0:00 1.41% httpd
26784 80 1 -4 0 79892K 10036K devfs 0 0:00 1.41% httpd
26785 80 1 -4 0 81304K 11228K devfs 0 0:00 1.41% httpd
26763 80 1 -4 0 83220K 13752K devfs 1 0:00 1.24% httpd
...etc...
As you can see there's plenty of free memory and the CPU is 70% idle
yet the load is sky high. When it's like this it's impossible to do
anything - every command can take anything from a few seconds to a few
minutes to respond - and the web user experience is shot to pieces. I
can't find a reference that explains what the devfs state indicates.
SA shows the following...
root at harold:~# sa
150107 242633.24re 1548.32cp 13avio 801k
31174 192785.02re 1488.13cp 32avio 502k httpd*
7047 6286.71re 21.53cp 67avio 2854k php
7703 164.83re 14.65cp 3avio 743k convert
8 19.82re 9.47cp 5016avio 1k gzip
7703 177.66re 7.70cp 2avio 1352k composite
4 18.90re 3.69cp 1avio 3k mysqldump
9 0.80re 0.76cp 23avio 144k bzip2
6 4.16re 0.53cp 22998avio 1731k find
2890 1126.31re 0.45cp 0avio 175988k perl*
5415 40.83re 0.32cp 11avio 123676k rateup
3009 14948.72re 0.32cp 3avio 9512k smtp
247 1812.85re 0.27cp 440avio 945k cleanup
7 163.69re 0.18cp 0avio 105k top
167 1809.71re 0.11cp 183avio 1117k pickup
2667 797.34re 0.07cp 0avio 29671k trivial-
rewrite
1820 96.40re 0.02cp 0avio 141901k ps
17 1482.98re 0.02cp 1avio 1083k sshd*
35629 6318.10re 0.01cp 0avio 18016816k sh
15232 391.68re 0.01cp 0avio 935430k sendmail
599 1223.30re 0.01cp 28avio 44065k bounce
7 0.95re 0.01cp 8avio 596k sa
35 1952.66re 0.01cp 11avio 16013k ***other
25 0.07re 0.01cp 18avio 1447k rm
15232 203.94re 0.00cp 7avio 2994436k postdrop
61 1.20re 0.00cp 5avio 34481k git
3 123.65re 0.00cp 12avio 796k mail
92 345.77re 0.00cp 0avio 34405k scache
31 2.13re 0.00cp 66avio 7115k newsyslog
6 2140.32re 0.00cp 14avio 6835k bash
5463 290.01re 0.00cp 0avio 2584397k grep
5 1479.06re 0.00cp 17avio 3878k sshd
1830 99.75re 0.00cp 0avio 1221440k wc
4 2.86re 0.00cp 2avio 13472k less
3 0.46re 0.00cp 0avio 10048k pgrep
68 27.44re 0.00cp 0avio 69632k sh*
3 5.06re 0.00cp 1avio 13120k showq
59 109.55re 0.00cp 0avio 243008k smtpd
183 0.54re 0.00cp 2avio 518400k unlink
2 1.61re 0.00cp 33avio 4608k whereis
12 0.00re 0.00cp 0avio 35000k adjkerntz
50 189.34re 0.00cp 0avio 165000k anvil
364 1.27re 0.00cp 0avio 947300k atrun
3 0.00re 0.00cp 0avio 53100k awk
10 0.05re 0.00cp 0avio 298800k bash*
27 4.15re 0.00cp 0avio 62000k cat
7 0.00re 0.00cp 0avio 44800k cmp
40 0.01re 0.00cp 2avio 79100k cp
2937 5227.93re 0.00cp 0avio 5143200k cron*
6 0.00re 0.00cp 0avio 22800k date
183 14.68re 0.00cp 2avio 578900k dd
5 0.00re 0.00cp 0avio 10600k egrep
132 241.91re 0.00cp 0avio 552400k error
6 0.05re 0.00cp 0avio 0k expr
245 444.67re 0.00cp 1avio 1181500k flush
3 0.00re 0.00cp 0avio 0k hostname
8 0.03re 0.00cp 0avio 21000k id
4 0.04re 0.00cp 0avio 173800k ipcs
183 0.49re 0.00cp 0avio 670000k jot
16 0.35re 0.00cp 1avio 65900k ls
3 1.52re 0.00cp 11avio 16300k man
2 0.06re 0.00cp 3avio 6800k manpath
10 0.00re 0.00cp 2avio 29200k mktemp
2 0.00re 0.00cp 0avio 0k mount
1287 4.64re 0.00cp 3avio 3717800k mv
4 0.12re 0.00cp 0avio 17600k postqueue
57 10.01re 0.00cp 0avio 242300k proxymap
7 4.19re 0.00cp 0avio 60500k sed
7 4.18re 0.00cp 0avio 10100k sort
7 0.05re 0.00cp 0avio 6100k sysctl
9 0.00re 0.00cp 0avio 15000k tee
4 26.15re 0.00cp 0avio 9300k time
2 0.48re 0.00cp 0avio 25400k zcat
I'm fairly new to SA but I can't see anything in there that indicates
a problem. If I'm missing something please let mw know.
If anyone has any ideas as to what might be causing this and how to
fix it I'd appreciate it. I'm getting close to giving up and simply
reinstalling it.
-Stut
--
http://stut.net/
More information about the freebsd-questions
mailing list