FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!

Wed Jul 18 22:40:05 UTC 2007

Hello everyone,

I am contacting -performance, -questions, and -hackers in the hope someone
helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4

uname:
FreeBSD 6.2-STABLE Fri Jun 22 12:17:03 UTC 2007 amd64

installed php modules:
php5-5.2.3 PHP Scripting Language (Apache Module and CLI)
php5-gd-5.2.3 The gd shared extension for php
php5-mysql-5.2.3 The mysql shared extension for php
php5-pcre-5.2.3 The pcre shared extension for php
php5-session-5.2.3 The session shared extension for php
php5-simplexml-5.2.3 The simplexml shared extension for php
php5-tokenizer-5.2.3 The tokenizer shared extension for php
php5-xml-5.2.3 The xml shared extension for php

apache version:
apache-2.2.4_2 Version 2.2 of Apache web server with prefork MPM.

system:
real memory = 5100273664 (4864 MB)
avail memory = 4120178688 (3929 MB)

CPU: Intel(R) Xeon(TM) CPU 2.66GHz (2666.78-MHz K8-class CPU)
Logical CPUs per core: 2
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs

The problem:

Right after starting apache, the loads on the server will climb to 10-40's
and the application will become unacceptably slow. This will go on until few
users are using the said application. (note: other servers running older
FreeBSD versions on dual cpus running the same code don't exhibit this
system% problem) top shows more than 60% of the CPU time is spent on system:

CPU states: 19.9% user, 0.0% nice, 73.7% system, 1.7% interrupt, 4.7% idle
Mem: 398M Active, 2226M Inact, 253M Wired, 202M Cache, 214M Buf, 567M Free

The apache processes look like:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
56882 www 1 103 0 139M 17516K select 0 0:03 12.66% httpd
56862 www 1 100 0 139M 21168K CPU2 6 0:06 11.87% httpd
56830 www 1 99 0 138M 19684K select 2 0:09 10.76% httpd
56887 www 1 105 0 139M 17488K select 6 0:01 10.49% httpd
56852 www 1 99 0 138M 20352K select 4 0:06 10.26% httpd
56889 www 1 106 0 139M 17548K select 6 0:01 10.04% httpd
56894 www 1 109 0 139M 17024K select 6 0:01 9.79% httpd
56839 www 1 99 0 138M 21216K select 6 0:06 9.36% httpd
56866 www 1 99 0 138M 17664K select 6 0:04 9.36% httpd
56890 www 1 108 0 138M 16180K select 4 0:01 9.29% httpd
56848 www 1 99 0 138M 20460K select 2 0:06 9.27% httpd
56865 www 1 99 0 138M 18920K select 2 0:05 9.23% httpd
56883 www 1 102 0 138M 16744K select 4 0:02 8.99% httpd
56870 www 1 100 0 139M 18440K select 2 0:03 8.86% httpd
56850 www 1 98 0 138M 21284K select 6 0:05 8.84% httpd
56860 www 1 99 0 138M 19584K select 0 0:05 8.70% httpd
56864 www 1 99 0 139M 18028K select 2 0:04 8.23% httpd
56854 www 1 99 0 138M 20696K select 6 0:05 8.23% httpd
56853 www 1 98 0 138M 19564K select 4 0:06 8.11% httpd
56835 www 1 98 0 139M 20276K CPU6 4 0:07 8.10% httpd
56849 www 1 98 0 138M 19532K select 0 0:05 7.95% httpd
56851 www 1 98 0 139M 20252K select 4 0:05 7.35% httpd
56888 www 1 4 0 139M 17100K sbwait 6 0:01 7.31% httpd
56869 www 1 100 0 139M 18632K select 4 0:02 6.75% httpd
56861 www 1 98 0 139M 18404K select 0 0:04 6.58% httpd
56863 www 1 98 0 139M 20220K select 2 0:03 6.40% httpd
56867 www 1 99 0 138M 17452K select 6 0:03 6.39% httpd
56868 www 1 99 0 138M 18376K select 0 0:03 6.20% httpd
56893 www 1 107 0 138M 12964K select 0 0:00 5.62% httpd
56878 www 1 100 0 138M 16732K select 6 0:02 5.27% httpd
56881 www 1 100 0 138M 16288K select 6 0:01 2.18% httpd

I had to lower MaxClients on apache substancially from 128 to 32, or loads
would quickly go to 40+. (Other servers with dual cpus instead of quad and
apache 1.3 on freebsd 6.0 don't have this problem)

vmstat 1:
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id
0 1 0 1380860 787212 1365 0 0 0 1312 1 0 0 486 559 842 13 22 65
1 1 0 1384588 787128 2724 0 0 0 2581 0 0 88 3038 82956 48776 19 38 43
4 1 0 1399232 782936 3328 0 0 0 2112 0 0 97 3592 101093 66497 24 50 26
0 1 2 1400200 781628 3726 0 0 0 2910 0 0 99 3529 100289 81531 23 58 19
19 1 0 1404000 778556 2263 0 0 0 1141 0 0 62 2964 73572 101432 19 76 5
15 1 1 1402452 776800 2499 0 0 0 1714 0 7 74 2965 68441 102276 19 78 3
15 1 0 1401548 777112 2213 0 0 0 2103 0 0 42 2491 105584 109418 15 79 6
8 1 1 1403324 778856 2606 0 0 0 2748 0 0 84 2996 75288 91676 22 76 2
0 1 3 1396864 781344 2764 0 0 0 3010 0 0 86 3393 90765 85952 25 70 5
1 2 0 1395520 782604 2774 0 0 0 2978 0 0 79 3195 88251 92623 20 63 17
6 1 0 1396096 781832 2641 0 0 0 2195 0 1 82 3347 96322 55942 21 42 37

iostat 1:
tty ad4 ad6 ad8 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 28 13.94 4 0.06 16.13 48 0.75 13.94 4 0.06 13 0 21 1 65
0 231 0.00 0 0.00 16.00 68 1.06 0.00 0 0.00 19 0 74 1 5
0 77 0.00 0 0.00 16.00 90 1.40 0.00 0 0.00 17 0 77 2 4
0 77 0.50 1 0.00 16.00 46 0.72 0.50 1 0.00 14 0 82 1 4
0 77 0.00 0 0.00 16.00 83 1.30 0.00 0 0.00 21 0 65 2 12
0 77 0.00 0 0.00 16.00 37 0.58 0.00 0 0.00 18 0 76 1 5
0 77 0.00 0 0.00 16.00 82 1.28 0.00 0 0.00 20 0 74 2 4
0 77 0.00 0 0.00 16.00 68 1.06 0.00 0 0.00 21 0 47 2 30
0 77 0.00 0 0.00 16.00 61 0.95 0.00 0 0.00 20 0 33 1 46
0 77 0.00 0 0.00 16.00 71 1.11 0.00 0 0.00 21 0 44 2 33
0 77 0.00 0 0.00 16.00 62 0.97 0.00 0 0.00 22 0 46 2 30

The kernel is custom built, with the following added (relevant) options:
options PMAP_SHPGPERPROC=4096
options ACCEPT_FILTER_HTTP

/boot/loader.conf:
kern.maxproc=32768
kern.ipc.shmmni=4096
kern.ipc.shmseg=2048
kern.ipc.semmns=960
kern.ipc.semmni=160
kern.ipc.semume=160
kern.ipc.semmnu=480

mysqld is also running on this server, altough it never goes past 8-10% CPU
usage. It is separated from apache on I/O, via jails and gmirror:
mirror/gm0 COMPLETE ad4
ad8
mirror/gm1 COMPLETE ad6
ad10

Now this web application isn't the best code out there, but this is a quad
cpu server and it's performing a lot worse than some servers I have running
with 6.0 with apache 1.3 for over 400 days.

Am I the only one getting terrible performance with apache2 on FreeBSD 6 ?

p.s: cc me, I am not subscribed to any of the lists.

Regards,

Mark