CPU usage 100% but no process hogging CPU

[Gunther Mayer]

Hi there,

I'm having some capacity issues on the FreeBSD 6.2/Core 2 Duo/2GB RAM 
server that I manage. For quite a few days now it constantly shows load 
averages of around 1 and a CPU usage of around 100%. Yet summing up the 
CPU usage of the individual processes running I hardly ever get to more 
than 5%, regardless of how long I watch top.

A snapshot of my top output looks like this:

last pid: 96102;  load averages:  1.28,  1.15,  
1.06                                                                                
up 22+08:33:16  13:55:03
122 processes: 2 running, 119 sleeping, 1 zombie
CPU states: 67.3% user,  0.0% nice, 32.7% system,  0.0% interrupt,  0.0% 
idle
Mem: 474M Active, 974M Inact, 186M Wired, 68M Cache, 213M Buf, 93M Free
Swap: 4064M Total, 4064M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  635 root        1 122    0 27304K  2644K select 656:38  1.27% syslog-ng
  844 www        20  20    0   411M   300M kserel 360:13  0.00% java
  837 user1       3  20    0 29048K  5672K kserel  34:30  0.00% radiusd
  788 pgsql       1  96    0 13516K  3824K select  10:03  0.00% postgres
  785 pgsql       1 115    0   120M  7436K select   9:02  0.00% postgres
  787 pgsql       1   8    0   120M 41112K nanslp   5:15  0.00% postgres

syslog-ng is quite busy as I use it to capture logs of more than 50 
remote sites. I have lots of slow queries in my postgres logs that I 
think are related to this bottleneck, though unoptimised queries and an 
ever growing amount of data are more likely to take the blame for that. 
High disk I/O in this regard could explain the high system utilisation, 
however.

I found out that I've been bitten by the freebsd-update bug 
(http://security.freebsd.org/advisories/FreeBSD-EN-07:05.freebsd-update.asc) 
which replaced my SMP kernel with a GENERIC one and I'm taking 
corrective action early tomorrow morning, but surely even with just a 
single CPU the load average should never be as high?

Where are those phantom CPU hogging processes?

Gunther