What is loading my server so much?

[Michael Powell]

Laszlo Nagy wrote:

> System is FreeBSD shopzeus.com 8.1-STABLE FreeBSD 8.1-STABLE #0: Sun Oct
> 31 02:55:28 EDT 2010     amd64
> It has two quad-core Xeon CPUs, 24GB memory, and a RAID 1+0 array with
> 10 disks + Areca 1680 controller with 2GB write back cache.
> 
> Server is running: mailscanner + apache multihost + PHP + postgresql.
> Main load on the server is usually postgresql.
> 
> Today something happened. Number of http processes went up to 200. As a
> result, number of connections to database also went up to 200, and the
> web server is now refusing clients with "Cannot connect to database"
> messages (coming from PHP).
> 
> This is a typical output from top:
> 
> last pid: 12789;  load averages:  7.77, 10.77,
> 13.46
> up 26+03:00:30  06:22:04
> 6637 processes: 7 running, 623 sleeping, 7 zombie
[snip]                                     ^^^^^^^^^^^^^^^^^^^

First curiosity I would have is these 7 zombies. Which processes are they 
and how did they get into that state? Might give you a thread to pull on for 
unraveling. Also, there is some degree of discrepancy present wrt to 6 
thousand processes with only 7 running and 623 asleep. I suspect this is 
just top not getting it right. But it also looks like a forkbomb.

-Mike