What's using my system?

[Dan Nelson]

In the last episode (Jan 01), Kirk Strauser said:
> I'm staring at "top" running on my 6.0-STABLE system.  It's my
> desktop machine, and also serves a light web/mail load (including a
> few jails). Basically, the system should be idle about 99% of the
> time when I'm not actively doing something on it.  And yet it's not.
> 
> The CPU never gets above about 75% idle, but top never shows any
> processes doing much of anything.  The "last pid" field only rarely
> increases, so I'm relatively sure there's no processing forking off
> children that die too quickly for top to notice them.
> 
> So, what could be using my CPU?  And other than top and tailing
> various logfiles, what tools could help me find out?
> 
> Example top output:
> 
> last pid: 72475;  load averages:  0.26,  0.80,  1.69 up 28+21:59:16  13:26:30
> 214 processes: 4 running, 210 sleeping
> CPU states: 14.7% user,  0.0% nice, 10.5% system,  3.9% interrupt, 70.9% idle
> Mem: 752M Active, 186M Inact, 219M Wired, 63M Cache, 112M Buf, 23M Free
> Swap: 4096M Total, 161M Used, 3934M Free, 3% Inuse
> 
>   PID USERNAME         THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
> 61660 root               1  96    0 89288K 74624K select  37:50  1.71% Xorg
> 61791 kirk               1  96    0 34704K 25600K select  12:17  1.12% kdeinit
> 61822 kirk               1  96    0 40616K 28980K select  14:55  1.03% kdeinit
> 68111 kirk               1  96    0 33584K 23292K select   0:16  0.15% kdeinit
>  1525 ldap               3  20    0   134M  7136K kserel  80:59  0.00% slapd
> 61811 kirk               3  20  -76 16516K 10860K kserel  76:40  0.00% artsd
>  1442 mysql              5  20    0 59576K  2752K kserel  44:14  0.00% mysqld
>  1449 nagios             3  20    0  3900K  1228K kserel  37:36  0.00% nagios

My bets are on the above four processes.  libpthread threads do not report
%CPU usage, but those programs have definitely been consuming cpu since
the TIME column is large.  Switch to libc_r or libthr (you can use
libmap.conf to redirect individual programs) to get useful CPU stats. 
Artsd's cpu usage is very suspicious, since it shouldn't be doing
anything but sending sound samples to /dev/pcm.

-- 
	Dan Nelson
	dnelson at allantgroup.com