High load, lots of free memory and processes in devfs state
Stut
stuttle at gmail.com
Mon Jun 16 21:28:09 UTC 2008
On 16 Jun 2008, at 22:20, Patrick C wrote:
> Is the MySQL daemon still running on that box? I see a mysqldump but
> no mysqld. If it is, try doing a shutdown and see if the load
> decreases.
>
> Sounds odd, but I have been having similar issues with MySQL.
It's not, no. It used to but it doesn't run now. The mysqldump is a
nightly backup.
-Stut
> 2008/6/16 Stut <stuttle at gmail.com>:
>> On 16 Jun 2008, at 18:48, Oliver Fromme wrote:
>>>
>>> Stut <stuttle at gmail.com> wrote:
>>>>
>>>> I'm having a serious problem with one of my FreeBSD servers. It
>>>> runs
>>>> FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the
>>>> hardware and it's all OK. There's approximately 400 Apache
>>>> processes
>>>> running and a 2GB memcached instance.
>>>>
>>>> Up until last weekend this server was working perfectly at a level
>>>> of HTTP traffic higher than it's currently getting. Last week our
>>>> database server (separate server) died and had to be rebuilt. While
>>>> this was being done this server hosted the database. This has now
>>>> been completed and the PHP app is pointing back at the dedicated DB
>>>> server.
>>>>
>>>> Top shows the following...
>>>>
>>>> last pid: 26838; load averages: 10.22, 14.06, 13.55 up
>>>> 2+00:34:47 18:03:43
>>>> 619 processes: 1 running, 618 sleeping
>>>> CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt,
>>>> 70.0% idle
>>>> Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf,
>>>> 1747M Free
>>>> Swap: 8192M Total, 124K Used, 8192M Free
>>>>
>>>> PID UID THR PRI NICE SIZE RES STATE C TIME WCPU
>>>> COMMAND
>>>> 26807 80 1 -4 0 79892K 9996K devfs 0 0:00 2.81%
>>>> httpd
>>>> 26797 80 1 -4 0 82376K 12592K devfs 1 0:00 2.19%
>>>> httpd
>>>> 26791 80 1 -4 0 82376K 12636K devfs 0 0:00 1.85%
>>>> httpd
>>>> 26783 80 1 -4 0 82392K 12640K devfs 3 0:00 1.84%
>>>> httpd
>>>> [...]
>>>
>>> Please let "vmstat 5" run for a minute ... Anything
>>> that looks unusual?
>>
>> procs memory page disks
>> faults cpu
>> r b w avm fre flt re pi po fr sr da0 pa0 in sy
>> cs us
>> sy id
>> 1 412 2 8910132 2743568 2230 3 1 0 2106 93 0 0 1098 4733
>> 3025 4
>> 17 79
>> 1 412 0 8841792 2760548 1630 0 0 0 2276 0 21 0 658 3970
>> 6374 3
>> 26 71
>> 0 422 0 8804744 2765076 1349 0 0 0 1394 0 12 0 576 3454
>> 5131 3
>> 26 71
>> 1 421 0 8756808 2778660 1095 0 0 0 1581 0 48 0 574 3076
>> 4415 3
>> 25 72
>> 0 420 0 8684932 2800128 2049 0 0 0 2927 0 45 0 505 2998
>> 3770 3
>> 24 73
>> 1 380 0 8603676 2838452 1149 0 0 0 2842 0 22 0 505 3497
>> 4339 2
>> 26 72
>> 1 21 0 8280260 3097392 7630 0 0 0 19969 0 29 0 1241 8895
>> 6950 10
>> 30 60
>> 1 4 0 8195828 3131992 9113 0 0 0 9742 0 78 0 996 9405
>> 2830 7
>> 19 74
>> 0 20 0 8169288 3121688 2899 0 0 0 2098 0 47 0 1193 7740
>> 2733 4
>> 23 73
>> 11 13 0 8123476 3128688 1947 0 0 0 2160 0 52 0 1161 6231
>> 2617 4
>> 26 70
>> 1 14 0 8067304 3143984 2298 0 0 0 2885 0 57 0 1806 5572
>> 3373 4
>> 25 70
>> 2 17 1 8015924 3156168 2702 0 0 0 3164 0 23 0 1384 8243
>> 2770 6
>> 25 69
>> 1 22 0 7956476 3176376 2013 0 0 0 2879 0 23 0 917 7063
>> 2484 6
>> 25 69
>> 0 35 0 7944760 3150452 4274 0 0 0 2806 0 21 0 1591 8281
>> 3399 7
>> 25 68
>> 1 67 3 7903160 3158776 2043 0 0 0 2442 0 20 0 1095 6405
>> 3605 6
>> 25 70
>> 44 69 0 7872504 3147192 2569 0 0 0 1712 0 81 0 1137 5773
>> 4998 6
>> 26 69
>> 1 146 0 7849632 3135388 2095 0 0 0 1274 0 19 0 869 5550
>> 5466 5
>> 26 69
>> 3 195 2 7825932 3122116 2482 0 0 0 1586 0 15 0 863 5558
>> 6135 5
>> 26 69
>> 1 244 3 7798148 3111624 1609 0 0 0 1226 0 10 0 849 4027
>> 6477 4
>> 27 69
>> 2 273 3 7776772 3102948 2080 0 0 0 1310 0 14 0 604 4376
>> 6527 4
>> 26 70
>> 1 312 2 7768232 3079012 3148 0 0 0 1800 0 12 0 1066 6088
>> 10242 6
>> 28 66
>> 1 340 0 7742680 3068244 2020 0 0 0 1421 0 74 0 729 4407
>> 8204 4
>> 26 70
>> 2 366 0 7740324 3068068 1612 0 0 0 1553 0 11 0 613 3526
>> 6728 3
>> 26 70
>> 1 397 1 7928688 3059900 1886 0 0 0 1344 0 12 0 515 3081
>> 6864 3
>> 27 70
>> 1 400 0 8074560 3008988 4771 0 0 0 1457 0 14 0 950 5309
>> 8996 5
>> 27 68
>>
>> Lots of processes blocked - which I guess is what the devfs state
>> indicates.
>>
>>> Have you checked dmesg?
>>
>> Yes. The only odd thing I can see is the following message, but
>> from what
>> I've read it's not critical until you get 5 and it's only in there
>> once.
>>
>> "collecting pv entries -- suggest increasing PMAP_SHPGPERPROC|
>>
>>
>>> Is this FreeBSD i386 (32bit) or amd64 (64bit)?
>>
>> FreeBSD harold.freeads.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE #0:
>> Fri Jan 12
>> 08:43:30 UTC 2007 root at portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/
>> SMP
>> amd64
>>
>>> Have you considered updating? 6.2-RELEASE isn't the
>>> freshest anymore. You might even consider going to
>>> 7-stable and using the new ULE scheduler which copes
>>> better with SMP servers.
>>
>> I have, but I'd rather understand what's happening.
>>
>>>> As you can see there's plenty of free memory and the CPU is 70%
>>>> idle
>>>> yet the load is sky high.
>>>
>>> Well, load 10 isn't that much for a 4-way SMP system.
>>
>> A couple of weeks ago this server was fairly fast, load never
>> really going
>> beyond 3 and everything was reasonably responsive. Now it regularly
>> goes up
>> to and beyond a load of 10 (more often than not at the moment) and
>> everything is slow. This means our website users are getting a very
>> poor
>> experience which is reflected in our traffic levels which have
>> dropped by
>> about 25% since this started happening.
>>
>> Thanks for your help. Any other ideas?
>>
>> -Stut
>> _______________________________________________
>> freebsd-questions at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org
>> "
>>
More information about the freebsd-questions
mailing list