2 x quad-core system is slower that 2 x dual core on FreeBSD

[Alexey Popov]

Hi.

Ivan Voras wrote:
  > Many people (including me) have run FreeBSD on machines like yours
> without such problems, so let's dig further.
> 
> You don't have WITNESS, INVARIANTS, DIAGNOSTICS or something similar
> enabled? Can you try a generic SMP kernel (called "SMP" in 6.x; the
> "GENERIC" in 7.x has SMP by default) and see how it works?
> 
> Can you disable SMP and try with only one CPU (on the 2xquad machine)?
> You can do it in loader.conf by setting kern.smp.disabled=1, or perhaps
> in BIOS. If there's a problem in some hardware or a driver, you'd still
> get a big load on sys time. You might also want to halt certain logical
> CPUs in the OS itself (see smp(4) man page) and see if there's a certain
> relationship between how many CPUs are running and what the sys load is.
Now I'm running yesterday's FreeBSD 7.0-BETA3 amd64 with GENERIC kernel. 
I rebuilt kernel and world with clean make.conf. Also I rebuilt Apache, 
PHP and eAccelerator from scratch. I tried APC as well. No success.

I tried 7-STABLE with UP kernel (GENERIC built without SMP config 
option). It works fine and can handle around 5-10 requests per second. 
It consumes %sys time is much less than %user time (see top output in 
attach). I.e. it seems to work good as a simple server with not so 
powerfull CPU.

After that I rebuilt with SMP GENERIC kernel and put on that server 2 
times more requests that UP could handle. For the first time it worked 
good. Then I increased load to 2.5 times more than UP. Immediately 
Apache child count increased to MaxClients (24), most of them in RUN 
state, and %sys became greater than %user (see attach). I think after 
some threshold of load FreeBSD is paying more CPU time to the management 
of running processes than to run them.

Also I tried to halt CPUs by machdep.hlt_cpus sysctl, but in that case 
%sys in top was still much greater than %user.

With best regards,
Alexey Popov
-------------- next part --------------
last pid:  1100;  load averages:  8.55,  5.20,  2.35                    up 0+00:05:39  18:59:52
48 processes:  22 running, 26 sleeping
CPU states:  5.9% user,  0.0% nice, 81.3% system,  0.0% interrupt, 12.8% idle
Mem: 245M Active, 14M Inact, 102M Wired, 108K Cache, 48M Buf, 3543M Free
Swap: 2048M Total, 2048M Free

  PID USERNAME   THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
 1093 www          1 105    0 94524K 39716K CPU1   7   0:09 34.41% httpd
 1094 www          1 105    0 91452K 39732K select 4   0:09 34.01% httpd
 1097 www          1  -4    0    98M 48392K RUN    7   0:10 33.41% httpd
 1098 www          1 105    0 92476K 43176K CPU4   7   0:09 33.27% httpd
 1099 www          1  -4    0 92476K 40784K RUN    7   0:09 33.21% httpd
 1100 www          1  -4    0 92476K 41080K RUN    4   0:09 32.87% httpd
 1095 www          1  -4    0 92476K 40824K RUN    6   0:09 32.74% httpd
 1090 www          1  -4    0 96572K 42700K RUN    5   0:09 32.54% httpd
 1089 www          1  -4    0 93504K 42032K RUN    7   0:09 32.41% httpd
 1091 www          1  -4    0 95548K 44900K RUN    4   0:09 31.95% httpd
 1096 www          1  -4    0 98620K 47160K RUN    6   0:09 31.86% httpd
 1086 www          1  -4    0 96572K 45752K RUN    6   0:10 30.92% httpd
 1087 www          1 104    0 92476K 41016K CPU7   6   0:09 30.70% httpd
 1088 www          1 104    0 92476K 38332K CPU2   6   0:10 30.51% httpd
 1085 www          1 105    0 97596K 44416K CPU5   5   0:09 30.23% httpd
 1092 www          1  -4    0 92476K 40172K RUN    6   0:08 29.45% httpd
-------------- next part --------------
last pid:  2203;  load averages:  9.71, 10.08,  7.38                    up 0+00:17:48  18:50:39
35 processes:  5 running, 30 sleeping
CPU states: 82.2% user,  0.0% nice, 13.8% system,  0.0% interrupt,  4.0% idle
Mem: 128M Active, 15M Inact, 109M Wired, 132K Cache, 88M Buf, 3657M Free
Swap: 2048M Total, 2048M Free

  PID USERNAME   THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 2201 www          1 117    0 92476K 40352K RUN      0:02 10.48% httpd
 2203 www          1 116    0 96572K 40912K RUN      0:02 10.05% httpd
 2195 www          1 117    0 93500K 43012K select   0:02  9.61% httpd
 2202 www          1  20    0 91452K 41056K lockf    0:02  8.95% httpd
 2194 www          1 117    0   102M 49440K RUN      0:03  8.10% httpd
 2192 www          1 114    0 99648K 46168K select   0:03  6.76% httpd
 2179 www          1  20    0 92476K 41672K lockf    0:04  6.69% httpd
 2173 www          1 118    0   100M 48920K RUN      0:04  5.92% httpd
 2174 www          1  20    0 92476K 42964K lockf    0:04  5.67% httpd
  878 llp          1  96    0 32928K  4576K select   0:00  0.00% sshd
  891 root         1  20    0  9616K  2924K pause    0:00  0.00% csh
  691 root         1  96    0  8952K  2528K select   0:00  0.00% ntpd
 2161 root         1 131    0 86332K 13080K select   0:00  0.00% httpd
 2178 root         1  96    0  7656K  2168K RUN      0:00  0.00% top
  875 root         1   4    0 32928K  4512K sbwait   0:00  0.00% sshd
  774 root         1   4    0  4852K  1652K kqread   0:00  0.00% master