CPU usage 100% but no process hogging CPU
kris at FreeBSD.org
Sat Oct 27 05:46:43 PDT 2007
Gunther Mayer wrote:
> Hi there,
> I'm having some capacity issues on the FreeBSD 6.2/Core 2 Duo/2GB RAM
> server that I manage. For quite a few days now it constantly shows load
> averages of around 1 and a CPU usage of around 100%. Yet summing up the
> CPU usage of the individual processes running I hardly ever get to more
> than 5%, regardless of how long I watch top.
> A snapshot of my top output looks like this:
> last pid: 96102; load averages: 1.28, 1.15,
> up 22+08:33:16 13:55:03
> 122 processes: 2 running, 119 sleeping, 1 zombie
> CPU states: 67.3% user, 0.0% nice, 32.7% system, 0.0% interrupt, 0.0%
> Mem: 474M Active, 974M Inact, 186M Wired, 68M Cache, 213M Buf, 93M Free
> Swap: 4064M Total, 4064M Free
> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
> 635 root 1 122 0 27304K 2644K select 656:38 1.27% syslog-ng
> 844 www 20 20 0 411M 300M kserel 360:13 0.00% java
> 837 user1 3 20 0 29048K 5672K kserel 34:30 0.00% radiusd
> 788 pgsql 1 96 0 13516K 3824K select 10:03 0.00% postgres
> 785 pgsql 1 115 0 120M 7436K select 9:02 0.00% postgres
> 787 pgsql 1 8 0 120M 41112K nanslp 5:15 0.00% postgres
> syslog-ng is quite busy as I use it to capture logs of more than 50
> remote sites. I have lots of slow queries in my postgres logs that I
> think are related to this bottleneck, though unoptimised queries and an
> ever growing amount of data are more likely to take the blame for that.
> High disk I/O in this regard could explain the high system utilisation,
> I found out that I've been bitten by the freebsd-update bug
> which replaced my SMP kernel with a GENERIC one and I'm taking
> corrective action early tomorrow morning, but surely even with just a
> single CPU the load average should never be as high?
> Where are those phantom CPU hogging processes?
A couple of points:
1) top -S will show what the kernel is doing, which may be relevant.
2) Because it only samples once a second (by default), top is bad for
monitoring of any short-lived processes that may be using CPU for brief
periods and then exiting. Don't know if you have any on this workload
3) In 6.x threaded applications do not generate CPU usage data in top. ,
i.e. java is probably using more than 0% of your CPU :) I think this is
fixed in 7.0 and maybe also with libthr. Chances are you want to use
libthr even in 6.x for performance reasons (libkse has attrocious
performance). Use libmap.conf to switch the libraries.
More information about the freebsd-questions