Big Problem, Load Avg Very High

Dan Nelson dnelson at allantgroup.com
Fri Nov 5 22:08:18 PST 2004


In the last episode (Nov 06), Steven Adams said:
> We host a couple or sites on this server (not very big sites). My
> server load is always around 0.90 - 3.40. Sometimes it will jump up
> to 10-15.
> 
> At random it will jump up to 30-40 load and I wont even be able to
> get to the server, typing commands on the remote ip based kvm is VERY
> slow sometimes missing letters. As soon as im able to get top running
> it shows
> 
> ====================================
> 
> last pid: 52614;  load averages:  6.82, 15.75, 15.18 up 1+03:07:12  16:32:22
> 
> 462 processes: 1 running, 460 sleeping, 1 zombie
> CPU states:  0.0% user,  0.0% nice,  0.6% system,  0.6% interrupt, 98.7% idle
> 
> Mem: 615M Active, 68M Inact, 288M Wired, 29M Cache, 112M Buf, 1844K Free
> Swap: 1536M Total, 555M Used, 981M Free, 36% Inuse, 12K In

500MB of swap used?  You might have a process that's allocating too
much memory and causing the rest of the processes to swap to disk. Try
keeping a top session running all the time so you can monitor swap
usage and see if you notice any processes taking more memory than they
should.
 
> I did notice once when running systat -vmstat the amr0(scsi raid) jumps up
> to 99% busy copying 2-3MB/s for a few moments then goes back down..

That could be either regular disk activity or swap thrashing.  "vmstat
1" will tell you (watch the fre, pi and po columns).
 
> After one of the times it went to 50load it got this on the console
> screen. FYI: amrd0s1h is /home partition
> 
> Swap_pager: indefinite wait buffer: device: amrd0s1h, blkno: 103776, size: 32768
> Swap_pager: indefinite wait buffer: device: amrd0s1h, blkno: 130801, size: 4096

Most likely you're thrashing.  I've seen a couple other people mention
this error with 5.2.1, but not lately, so chances are 5.3 has fixed
this particular problem.

> It seems that its copying a lot of information to the swap drive and
> is running out of ram, which I don't know why it seems apache is
> taking up all of the ram for some weird reason?

Not weird at all.  If you are using perl or php modules, they can
really suck up ram if you get a lot of page hits at once.  You might
want to look at using fastcgi to separate perl/php from the apache
process itself.

-- 
	Dan Nelson
	dnelson at allantgroup.com


More information about the freebsd-questions mailing list