High load, lots of free memory and processes in devfs state

Mon Jun 16 19:17:17 UTC 2008

On 16 Jun 2008, at 18:48, Oliver Fromme wrote:
> Stut <stuttle at gmail.com> wrote:
>> I'm having a serious problem with one of my FreeBSD servers. It runs
>> FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the
>> hardware and it's all OK. There's approximately 400 Apache processes
>> running and a 2GB memcached instance.
>>
>> Up until last weekend this server was working perfectly at a level
>> of HTTP traffic higher than it's currently getting. Last week our
>> database server (separate server) died and had to be rebuilt. While
>> this was being done this server hosted the database. This has now
>> been completed and the PHP app is pointing back at the dedicated DB
>> server.
>>
>> Top shows the following...
>>
>> last pid: 26838;  load averages: 10.22, 14.06, 13.55         up
>> 2+00:34:47  18:03:43
>> 619 processes: 1 running, 618 sleeping
>> CPU states:  4.9% user,  0.0% nice, 24.8% system,  0.4% interrupt,
>> 70.0% idle
>> Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf,
>> 1747M Free
>> Swap: 8192M Total, 124K Used, 8192M Free
>>
>> PID    UID    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
>> COMMAND
>> 26807     80      1  -4    0 79892K  9996K devfs  0   0:00  2.81%
>> httpd
>> 26797     80      1  -4    0 82376K 12592K devfs  1   0:00  2.19%
>> httpd
>> 26791     80      1  -4    0 82376K 12636K devfs  0   0:00  1.85%
>> httpd
>> 26783     80      1  -4    0 82392K 12640K devfs  3   0:00  1.84%
>> httpd
>> [...]
>
> Please let "vmstat 5" run for a minute ...  Anything
> that looks unusual?

  procs      memory       page                    disks      
faults      cpu
  r b   w avm     fre     flt  re  pi  po  fr   sr da0 pa0   in   sy   
cs  us sy id
  1 412 2 8910132 2743568 2230  3   1   0 2106  93   0   0 1098 4733  
3025  4 17 79
  1 412 0 8841792 2760548 1630  0   0   0 2276   0  21   0  658 3970  
6374  3 26 71
  0 422 0 8804744 2765076 1349  0   0   0 1394   0  12   0  576 3454  
5131  3 26 71
  1 421 0 8756808 2778660 1095  0   0   0 1581   0  48   0  574 3076  
4415  3 25 72
  0 420 0 8684932 2800128 2049  0   0   0 2927   0  45   0  505 2998  
3770  3 24 73
  1 380 0 8603676 2838452 1149  0   0   0 2842   0  22   0  505 3497  
4339  2 26 72
  1 21  0 8280260 3097392 7630  0   0   0 19969  0  29   0 1241 8895  
6950 10 30 60
  1 4   0 8195828 3131992 9113  0   0   0 9742   0  78   0  996 9405  
2830  7 19 74
  0 20  0 8169288 3121688 2899  0   0   0 2098   0  47   0 1193 7740  
2733  4 23 73
11 13  0 8123476 3128688 1947  0   0   0 2160   0  52   0 1161 6231  
2617  4 26 70
  1 14  0 8067304 3143984 2298  0   0   0 2885   0  57   0 1806 5572  
3373  4 25 70
  2 17  1 8015924 3156168 2702  0   0   0 3164   0  23   0 1384 8243  
2770  6 25 69
  1 22  0 7956476 3176376 2013  0   0   0 2879   0  23   0  917 7063  
2484  6 25 69
  0 35  0 7944760 3150452 4274  0   0   0 2806   0  21   0 1591 8281  
3399  7 25 68
  1 67  3 7903160 3158776 2043  0   0   0 2442   0  20   0 1095 6405  
3605  6 25 70
44 69  0 7872504 3147192 2569  0   0   0 1712   0  81   0 1137 5773  
4998  6 26 69
  1 146 0 7849632 3135388 2095  0   0   0 1274   0  19   0  869 5550  
5466  5 26 69
  3 195 2 7825932 3122116 2482  0   0   0 1586   0  15   0  863 5558  
6135  5 26 69
  1 244 3 7798148 3111624 1609  0   0   0 1226   0  10   0  849 4027  
6477  4 27 69
  2 273 3 7776772 3102948 2080  0   0   0 1310   0  14   0  604 4376  
6527  4 26 70
  1 312 2 7768232 3079012 3148  0   0   0 1800   0  12   0 1066 6088  
10242 6 28 66
  1 340 0 7742680 3068244 2020  0   0   0 1421   0  74   0  729 4407  
8204  4 26 70
  2 366 0 7740324 3068068 1612  0   0   0 1553   0  11   0  613 3526  
6728  3 26 70
  1 397 1 7928688 3059900 1886  0   0   0 1344   0  12   0  515 3081  
6864  3 27 70
  1 400 0 8074560 3008988 4771  0   0   0 1457   0  14   0  950 5309  
8996  5 27 68

Lots of processes blocked - which I guess is what the devfs state  
indicates.

>  Have you checked dmesg?

Yes. The only odd thing I can see is the following message, but from  
what I've read it's not critical until you get 5 and it's only in  
there once.

"collecting pv entries -- suggest increasing PMAP_SHPGPERPROC|

> Is this FreeBSD i386 (32bit) or amd64 (64bit)?

FreeBSD harold.freeads.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri  
Jan 12 08:43:30 UTC 2007 root at portnoy.cse.buffalo.edu:/usr/obj/usr/src/ 
sys/SMP  amd64

> Have you considered updating?  6.2-RELEASE isn't the
> freshest anymore.  You might even consider going to
> 7-stable and using the new ULE scheduler which copes
> better with SMP servers.

I have, but I'd rather understand what's happening.

>> As you can see there's plenty of free memory and the CPU is 70% idle
>> yet the load is sky high.
>
> Well, load 10 isn't that much for a 4-way SMP system.

A couple of weeks ago this server was fairly fast, load never really  
going beyond 3 and everything was reasonably responsive. Now it  
regularly goes up to and beyond a load of 10 (more often than not at  
the moment) and everything is slow. This means our website users are  
getting a very poor experience which is reflected in our traffic  
levels which have dropped by about 25% since this started happening.

Thanks for your help. Any other ideas?

-Stut