no idle CPU ... system hogging it all ...

[David Schultz]

On Fri, Apr 11, 2003, Marc G. Fournier wrote:
> 
> Its been looking like this pretty much all day ... top shows nothing
> major, and the drive looks reaonably quiet ... there is nothing in
> messages to indicate a problem that I can see (even those enclosure
> messages have been reasonably quiet) ...

You can profile the kernel to find out, although at this hour I
probably can't recall all the necessary details.  I believe you
need to say 'make buildkernel CONFIGARGS=-p', then use kgmon(8)
and gprof(1) to extract and analyze the data.

Based on the ps output you gave, it looks like vmdaemon and syncer
are taking up most of the time.  This suggests that perhaps there
is a hardware or driver problem with disk I/O.  Some drivers
perform very small delays to wait for the hardware by spinning
instead of eating the overhead of a context switch.  It could be
that one of these drivers has a bug, or the hardware is taking
longer than expected to respond.

> What consumes SYS CPU?  Stuff like apache and jakarta-tomcat use up USER
> CPU, correct?

Every process consumes system time when it is running in the
kernel to process a trap or system call.  They use user time when
they are running their own (unprivileged) code.  Kernel threads
such as the pageout daemon and syncer use system time exclusively.

time(1) will show you how much system and user time a process
uses.  Unfortunately, you can't use it on a process that is
already running, and tools like 'ps' don't show both numbers
(although they can be easily hacked to do so.)  For example:

	> time du >/dev/null
	0.074u 0.568s 0:03.36 18.7%	11+285k 1431+0io 0pf+0w

This shows that the 'du' command took 74 ms of user time and 568 ms
in system calls.  If you're not using the C shell or you want more
descriptive output, use '/usr/bin/time -l'.