OS suddenly VERY busy

[John-Mark Gurney]

Mikhail T. wrote this message on Wed, Jul 20, 2005 at 00:19 -0400:
> After a couple of huge tarball extracts (`make extract' in jdk14 and jdk15)
> I noticed, things are a little slower. During the extracts, the mouse was
> moving with visible jerks. Indeed, the system seems VERY busy:
> 
>    11 users    Load  1.18  1.52  1.40                  Jul 20 00:14
> 
> Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER
>         Tot   Share      Tot    Share    Free         in  out     in  out
> Act 1060360  105932  1341624   145252  151016 count
> All 1750432  115908  8911872   174532         pages
>                                                           zfod   Interrupts
> Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    1277 total
>      4     1146      1819  345 443k 1640  672      266572 wire      4 irq1: atkb
				 ^^^^
well, I'd say 443k syscalls/time interval isn't doing nothing...

[...]

> The machine is idle and is not doing anything in user-space according to both
> top and vmstat's "pigs" display.

the problem is that your machine is sooooo fast that all of the
processes that are running are exiting before they can be observed
by pigs or top (or even accumulate enough cpu time to be worth
showing)...

> Yet it is noticably slower. Trying to compile something pushes the load above
> 2. What is it doing?
> 
> This is a single-CPU Opteron running:
> 
> 	FreeBSD 5.4-STABLE #0: Fri Jun 10 09:11:30 EDT 2005 amd64
> 
> The box has 2Gb of RAM, but NO SWAP.

run ps lax a few times, and notice which process is fork bombing your
box by seeing which process has the most changing children...  (i.e.
the ppid, 3rd column, of the process that isn't in the next run)...
sort -n +1 -2 + diff will help find which ones...

ps lax | sort -n +1 -2 > tmpa; sleep 2; ps lax | sort -n +1 -2 > tmpb; diff tmpa tmpb

look at the ppid (3rd column) of any new or missing processes, and
you probably have your culprit...

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."