CPU usage 100% but no process hogging CPU

[Bruce Cran]

Gunther Mayer wrote:
> Hi there,
> 
> I'm having some capacity issues on the FreeBSD 6.2/Core 2 Duo/2GB RAM 
> server that I manage. For quite a few days now it constantly shows load 
> averages of around 1 and a CPU usage of around 100%. Yet summing up the 
> CPU usage of the individual processes running I hardly ever get to more 
> than 5%, regardless of how long I watch top.
> 
> A snapshot of my top output looks like this:
> 
> last pid: 96102;  load averages:  1.28,  1.15,  
> 1.06                                                                                
> up 22+08:33:16  13:55:03
> 122 processes: 2 running, 119 sleeping, 1 zombie
> CPU states: 67.3% user,  0.0% nice, 32.7% system,  0.0% interrupt,  0.0% 
> idle
> Mem: 474M Active, 974M Inact, 186M Wired, 68M Cache, 213M Buf, 93M Free
> Swap: 4064M Total, 4064M Free
> 
>  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
>  635 root        1 122    0 27304K  2644K select 656:38  1.27% syslog-ng
>  844 www        20  20    0   411M   300M kserel 360:13  0.00% java
>  837 user1       3  20    0 29048K  5672K kserel  34:30  0.00% radiusd
>  788 pgsql       1  96    0 13516K  3824K select  10:03  0.00% postgres
>  785 pgsql       1 115    0   120M  7436K select   9:02  0.00% postgres
>  787 pgsql       1   8    0   120M 41112K nanslp   5:15  0.00% postgres
> 
> syslog-ng is quite busy as I use it to capture logs of more than 50 
> remote sites. I have lots of slow queries in my postgres logs that I 
> think are related to this bottleneck, though unoptimised queries and an 
> ever growing amount of data are more likely to take the blame for that. 
> High disk I/O in this regard could explain the high system utilisation, 
> however.
> 
> I found out that I've been bitten by the freebsd-update bug 
> (http://security.freebsd.org/advisories/FreeBSD-EN-07:05.freebsd-update.asc) 
> which replaced my SMP kernel with a GENERIC one and I'm taking 
> corrective action early tomorrow morning, but surely even with just a 
> single CPU the load average should never be as high?
> 
> Where are those phantom CPU hogging processes?
> 

By default top doesn't display system (kernel) processes, which can take 
  up lots of CPU time.  To show these, run top with the "-S" flag.

--
Bruce