High load, lots of free memory and processes in devfs state

[Patrick C]

Is the MySQL daemon still running on that box? I see a mysqldump but
no mysqld. If it is, try doing a shutdown and see if the load
decreases.

Sounds odd, but I have been having similar issues with MySQL.

-Patrick

2008/6/16 Stut <stuttle at gmail.com>:
> On 16 Jun 2008, at 18:48, Oliver Fromme wrote:
>>
>> Stut <stuttle at gmail.com> wrote:
>>>
>>> I'm having a serious problem with one of my FreeBSD servers. It runs
>>> FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the
>>> hardware and it's all OK. There's approximately 400 Apache processes
>>> running and a 2GB memcached instance.
>>>
>>> Up until last weekend this server was working perfectly at a level
>>> of HTTP traffic higher than it's currently getting. Last week our
>>> database server (separate server) died and had to be rebuilt. While
>>> this was being done this server hosted the database. This has now
>>> been completed and the PHP app is pointing back at the dedicated DB
>>> server.
>>>
>>> Top shows the following...
>>>
>>> last pid: 26838;  load averages: 10.22, 14.06, 13.55         up
>>> 2+00:34:47  18:03:43
>>> 619 processes: 1 running, 618 sleeping
>>> CPU states:  4.9% user,  0.0% nice, 24.8% system,  0.4% interrupt,
>>> 70.0% idle
>>> Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf,
>>> 1747M Free
>>> Swap: 8192M Total, 124K Used, 8192M Free
>>>
>>> PID    UID    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
>>> COMMAND
>>> 26807     80      1  -4    0 79892K  9996K devfs  0   0:00  2.81%
>>> httpd
>>> 26797     80      1  -4    0 82376K 12592K devfs  1   0:00  2.19%
>>> httpd
>>> 26791     80      1  -4    0 82376K 12636K devfs  0   0:00  1.85%
>>> httpd
>>> 26783     80      1  -4    0 82392K 12640K devfs  3   0:00  1.84%
>>> httpd
>>> [...]
>>
>> Please let "vmstat 5" run for a minute ...  Anything
>> that looks unusual?
>
>  procs      memory       page                    disks     faults      cpu
>  r b   w avm     fre     flt  re  pi  po  fr   sr da0 pa0   in   sy  cs  us
> sy id
>  1 412 2 8910132 2743568 2230  3   1   0 2106  93   0   0 1098 4733 3025  4
> 17 79
>  1 412 0 8841792 2760548 1630  0   0   0 2276   0  21   0  658 3970 6374  3
> 26 71
>  0 422 0 8804744 2765076 1349  0   0   0 1394   0  12   0  576 3454 5131  3
> 26 71
>  1 421 0 8756808 2778660 1095  0   0   0 1581   0  48   0  574 3076 4415  3
> 25 72
>  0 420 0 8684932 2800128 2049  0   0   0 2927   0  45   0  505 2998 3770  3
> 24 73
>  1 380 0 8603676 2838452 1149  0   0   0 2842   0  22   0  505 3497 4339  2
> 26 72
>  1 21  0 8280260 3097392 7630  0   0   0 19969  0  29   0 1241 8895 6950 10
> 30 60
>  1 4   0 8195828 3131992 9113  0   0   0 9742   0  78   0  996 9405 2830  7
> 19 74
>  0 20  0 8169288 3121688 2899  0   0   0 2098   0  47   0 1193 7740 2733  4
> 23 73
> 11 13  0 8123476 3128688 1947  0   0   0 2160   0  52   0 1161 6231 2617  4
> 26 70
>  1 14  0 8067304 3143984 2298  0   0   0 2885   0  57   0 1806 5572 3373  4
> 25 70
>  2 17  1 8015924 3156168 2702  0   0   0 3164   0  23   0 1384 8243 2770  6
> 25 69
>  1 22  0 7956476 3176376 2013  0   0   0 2879   0  23   0  917 7063 2484  6
> 25 69
>  0 35  0 7944760 3150452 4274  0   0   0 2806   0  21   0 1591 8281 3399  7
> 25 68
>  1 67  3 7903160 3158776 2043  0   0   0 2442   0  20   0 1095 6405 3605  6
> 25 70
> 44 69  0 7872504 3147192 2569  0   0   0 1712   0  81   0 1137 5773 4998  6
> 26 69
>  1 146 0 7849632 3135388 2095  0   0   0 1274   0  19   0  869 5550 5466  5
> 26 69
>  3 195 2 7825932 3122116 2482  0   0   0 1586   0  15   0  863 5558 6135  5
> 26 69
>  1 244 3 7798148 3111624 1609  0   0   0 1226   0  10   0  849 4027 6477  4
> 27 69
>  2 273 3 7776772 3102948 2080  0   0   0 1310   0  14   0  604 4376 6527  4
> 26 70
>  1 312 2 7768232 3079012 3148  0   0   0 1800   0  12   0 1066 6088 10242 6
> 28 66
>  1 340 0 7742680 3068244 2020  0   0   0 1421   0  74   0  729 4407 8204  4
> 26 70
>  2 366 0 7740324 3068068 1612  0   0   0 1553   0  11   0  613 3526 6728  3
> 26 70
>  1 397 1 7928688 3059900 1886  0   0   0 1344   0  12   0  515 3081 6864  3
> 27 70
>  1 400 0 8074560 3008988 4771  0   0   0 1457   0  14   0  950 5309 8996  5
> 27 68
>
> Lots of processes blocked - which I guess is what the devfs state indicates.
>
>>  Have you checked dmesg?
>
> Yes. The only odd thing I can see is the following message, but from what
> I've read it's not critical until you get 5 and it's only in there once.
>
> "collecting pv entries -- suggest increasing PMAP_SHPGPERPROC|
>
>
>> Is this FreeBSD i386 (32bit) or amd64 (64bit)?
>
> FreeBSD harold.freeads.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12
> 08:43:30 UTC 2007 root at portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP
>  amd64
>
>> Have you considered updating?  6.2-RELEASE isn't the
>> freshest anymore.  You might even consider going to
>> 7-stable and using the new ULE scheduler which copes
>> better with SMP servers.
>
> I have, but I'd rather understand what's happening.
>
>>> As you can see there's plenty of free memory and the CPU is 70% idle
>>> yet the load is sky high.
>>
>> Well, load 10 isn't that much for a 4-way SMP system.
>
> A couple of weeks ago this server was fairly fast, load never really going
> beyond 3 and everything was reasonably responsive. Now it regularly goes up
> to and beyond a load of 10 (more often than not at the moment) and
> everything is slow. This means our website users are getting a very poor
> experience which is reflected in our traffic levels which have dropped by
> about 25% since this started happening.
>
> Thanks for your help. Any other ideas?
>
> -Stut
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"
>