High load, lots of free memory and processes in devfs state
Stut
stuttle at gmail.com
Mon Jun 16 15:57:08 UTC 2008
Does anybody have any ideas about this problem?
So far today I've tried moving memcache off this machine, played with
a few settings in httpd.conf but nothing seems to make a difference.
Any help would be greatly appreciated. I'm rapidly losing what's left
of my hair!!
-Stut
On 15 Jun 2008, at 18:18, Stut wrote:
> Hi,
>
> I'm having a serious problem with one of my FreeBSD servers. It runs
> FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the
> hardware and it's all OK. There's approximately 400 Apache processes
> running and a 2GB memcached instance.
>
> Up until last weekend this server was working perfectly at a level
> of HTTP traffic higher than it's currently getting. Last week our
> database server (separate server) died and had to be rebuilt. While
> this was being done this server hosted the database. This has now
> been completed and the PHP app is pointing back at the dedicated DB
> server.
>
> Top shows the following...
>
> last pid: 26838; load averages: 10.22, 14.06, 13.55 up
> 2+00:34:47 18:03:43
> 619 processes: 1 running, 618 sleeping
> CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt,
> 70.0% idle
> Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf,
> 1747M Free
> Swap: 8192M Total, 124K Used, 8192M Free
>
> PID UID THR PRI NICE SIZE RES STATE C TIME WCPU
> COMMAND
> 26807 80 1 -4 0 79892K 9996K devfs 0 0:00 2.81%
> httpd
> 26797 80 1 -4 0 82376K 12592K devfs 1 0:00 2.19%
> httpd
> 26791 80 1 -4 0 82376K 12636K devfs 0 0:00 1.85%
> httpd
> 26783 80 1 -4 0 82392K 12640K devfs 3 0:00 1.84%
> httpd
> 26780 80 1 -4 0 79892K 10036K devfs 0 0:00 1.51%
> httpd
> 26801 80 1 -4 0 81292K 11104K devfs 3 0:00 1.50%
> httpd
> 26786 80 1 -4 0 80456K 10796K devfs 1 0:00 1.41%
> httpd
> 26784 80 1 -4 0 79892K 10036K devfs 0 0:00 1.41%
> httpd
> 26785 80 1 -4 0 81304K 11228K devfs 0 0:00 1.41%
> httpd
> 26763 80 1 -4 0 83220K 13752K devfs 1 0:00 1.24%
> httpd
> ...etc...
>
> As you can see there's plenty of free memory and the CPU is 70% idle
> yet the load is sky high. When it's like this it's impossible to do
> anything - every command can take anything from a few seconds to a
> few minutes to respond - and the web user experience is shot to
> pieces. I can't find a reference that explains what the devfs state
> indicates.
>
> SA shows the following...
>
> root at harold:~# sa
> 150107 242633.24re 1548.32cp 13avio 801k
> 31174 192785.02re 1488.13cp 32avio 502k httpd*
> 7047 6286.71re 21.53cp 67avio 2854k php
> 7703 164.83re 14.65cp 3avio 743k convert
> 8 19.82re 9.47cp 5016avio 1k gzip
> 7703 177.66re 7.70cp 2avio 1352k composite
> 4 18.90re 3.69cp 1avio 3k mysqldump
> 9 0.80re 0.76cp 23avio 144k bzip2
> 6 4.16re 0.53cp 22998avio 1731k find
> 2890 1126.31re 0.45cp 0avio 175988k perl*
> 5415 40.83re 0.32cp 11avio 123676k rateup
> 3009 14948.72re 0.32cp 3avio 9512k smtp
> 247 1812.85re 0.27cp 440avio 945k cleanup
> 7 163.69re 0.18cp 0avio 105k top
> 167 1809.71re 0.11cp 183avio 1117k pickup
> 2667 797.34re 0.07cp 0avio 29671k trivial-
> rewrite
> 1820 96.40re 0.02cp 0avio 141901k ps
> 17 1482.98re 0.02cp 1avio 1083k sshd*
> 35629 6318.10re 0.01cp 0avio 18016816k sh
> 15232 391.68re 0.01cp 0avio 935430k sendmail
> 599 1223.30re 0.01cp 28avio 44065k bounce
> 7 0.95re 0.01cp 8avio 596k sa
> 35 1952.66re 0.01cp 11avio 16013k ***other
> 25 0.07re 0.01cp 18avio 1447k rm
> 15232 203.94re 0.00cp 7avio 2994436k postdrop
> 61 1.20re 0.00cp 5avio 34481k git
> 3 123.65re 0.00cp 12avio 796k mail
> 92 345.77re 0.00cp 0avio 34405k scache
> 31 2.13re 0.00cp 66avio 7115k newsyslog
> 6 2140.32re 0.00cp 14avio 6835k bash
> 5463 290.01re 0.00cp 0avio 2584397k grep
> 5 1479.06re 0.00cp 17avio 3878k sshd
> 1830 99.75re 0.00cp 0avio 1221440k wc
> 4 2.86re 0.00cp 2avio 13472k less
> 3 0.46re 0.00cp 0avio 10048k pgrep
> 68 27.44re 0.00cp 0avio 69632k sh*
> 3 5.06re 0.00cp 1avio 13120k showq
> 59 109.55re 0.00cp 0avio 243008k smtpd
> 183 0.54re 0.00cp 2avio 518400k unlink
> 2 1.61re 0.00cp 33avio 4608k whereis
> 12 0.00re 0.00cp 0avio 35000k adjkerntz
> 50 189.34re 0.00cp 0avio 165000k anvil
> 364 1.27re 0.00cp 0avio 947300k atrun
> 3 0.00re 0.00cp 0avio 53100k awk
> 10 0.05re 0.00cp 0avio 298800k bash*
> 27 4.15re 0.00cp 0avio 62000k cat
> 7 0.00re 0.00cp 0avio 44800k cmp
> 40 0.01re 0.00cp 2avio 79100k cp
> 2937 5227.93re 0.00cp 0avio 5143200k cron*
> 6 0.00re 0.00cp 0avio 22800k date
> 183 14.68re 0.00cp 2avio 578900k dd
> 5 0.00re 0.00cp 0avio 10600k egrep
> 132 241.91re 0.00cp 0avio 552400k error
> 6 0.05re 0.00cp 0avio 0k expr
> 245 444.67re 0.00cp 1avio 1181500k flush
> 3 0.00re 0.00cp 0avio 0k hostname
> 8 0.03re 0.00cp 0avio 21000k id
> 4 0.04re 0.00cp 0avio 173800k ipcs
> 183 0.49re 0.00cp 0avio 670000k jot
> 16 0.35re 0.00cp 1avio 65900k ls
> 3 1.52re 0.00cp 11avio 16300k man
> 2 0.06re 0.00cp 3avio 6800k manpath
> 10 0.00re 0.00cp 2avio 29200k mktemp
> 2 0.00re 0.00cp 0avio 0k mount
> 1287 4.64re 0.00cp 3avio 3717800k mv
> 4 0.12re 0.00cp 0avio 17600k postqueue
> 57 10.01re 0.00cp 0avio 242300k proxymap
> 7 4.19re 0.00cp 0avio 60500k sed
> 7 4.18re 0.00cp 0avio 10100k sort
> 7 0.05re 0.00cp 0avio 6100k sysctl
> 9 0.00re 0.00cp 0avio 15000k tee
> 4 26.15re 0.00cp 0avio 9300k time
> 2 0.48re 0.00cp 0avio 25400k zcat
>
> I'm fairly new to SA but I can't see anything in there that
> indicates a problem. If I'm missing something please let mw know.
>
> If anyone has any ideas as to what might be causing this and how to
> fix it I'd appreciate it. I'm getting close to giving up and simply
> reinstalling it.
>
> -Stut
>
> --
> http://stut.net/
More information about the freebsd-questions
mailing list