High load, lots of free memory and processes in devfs state

[Oliver Fromme]

Stut <stuttle at gmail.com> wrote:
 > I'm having a serious problem with one of my FreeBSD servers. It runs  
 > FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the  
 > hardware and it's all OK. There's approximately 400 Apache processes  
 > running and a 2GB memcached instance.
 > 
 > Up until last weekend this server was working perfectly at a level  
 > of HTTP traffic higher than it's currently getting. Last week our  
 > database server (separate server) died and had to be rebuilt. While  
 > this was being done this server hosted the database. This has now  
 > been completed and the PHP app is pointing back at the dedicated DB  
 > server.
 > 
 > Top shows the following...
 > 
 > last pid: 26838;  load averages: 10.22, 14.06, 13.55         up  
 > 2+00:34:47  18:03:43
 > 619 processes: 1 running, 618 sleeping
 > CPU states:  4.9% user,  0.0% nice, 24.8% system,  0.4% interrupt,  
 > 70.0% idle
 > Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf,  
 > 1747M Free
 > Swap: 8192M Total, 124K Used, 8192M Free
 > 
 >  PID    UID    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU  
 > COMMAND
 > 26807     80      1  -4    0 79892K  9996K devfs  0   0:00  2.81%  
 > httpd
 > 26797     80      1  -4    0 82376K 12592K devfs  1   0:00  2.19%  
 > httpd
 > 26791     80      1  -4    0 82376K 12636K devfs  0   0:00  1.85%  
 > httpd
 > 26783     80      1  -4    0 82392K 12640K devfs  3   0:00  1.84%  
 > httpd
 > [...]

Please let "vmstat 5" run for a minute ...  Anything
that looks unusual?  Have you checked dmesg?
Is this FreeBSD i386 (32bit) or amd64 (64bit)?

Have you considered updating?  6.2-RELEASE isn't the
freshest anymore.  You might even consider going to
7-stable and using the new ULE scheduler which copes
better with SMP servers.

 > As you can see there's plenty of free memory and the CPU is 70% idle  
 > yet the load is sky high.

Well, load 10 isn't that much for a 4-way SMP system.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"Perl will consistently give you what you want,
unless what you want is consistency."
        -- Larry Wall