serious performance problems with 6.2 Release

[Steven H. Baeighkley]

Greetings,

We are having some bizarre performance problems on a freshly installed 
6.2 Release server. This is a supermicro superserver 6022c dual 2.0 Xeon 
with 2GB RAM. These CPUs do support hyperthreading. We have done 
significant testing with both hyperthreading turned on and off in the 
bios and in the OS, to no avail.

The server is configured as a web server with apache 2.2.4 php 5.2.0 and 
ZendOptimizer. We are running proftpd 1.3.1rc1 and perl 5.8.8. We have 
another server running 4.11 with the same exact hardware and software 
versions. We have updated to the newest bios that Supermicro provides.

The trouble is that the 6.2 box performs significantly worse than the 
4.11 server. The load on the 6.2 server is regularly between 2.0 and 
6.0. The load on the 4.11 server is between .57 and 1 despite often 
servicing more connections.

We began this process to upgrade into the 6 tree because 4 is EOL. We 
kept the old 4.11 drive from this machine and when replacing it into the 
box performance is excellent just like our other 4.11 box. We have tired 
multiple tuning variables as recommended by both FreeBSD and apache and 
tried the recommendations in the 6.2 errata as well. The 6.2 errata 
states that kern.ipc.nmbclusters="0" will help the kernel memory 
allocator properly deal with high network traffic. We tried this and 
initially thought that the box was showing wonderful performance, but 
then we realized that the box was not allowing much network access at 
all. A single ssh and proftpd connection were all it would accept. 
Apache wouldn't even start giving a MaxClients error. Removing this 
option returned it to functional though poor performance mode. Are we 
missing something with how to use this variable? IS this expected behavior?

This particular hardware does display some oddities on both machines, 
running either 6.2 or 4.11. We know that FreeBSD has hyperthreading 
turned off by default. We have done some additional testing with 
hyperthreading turned on in the OS, but we wish for it to remain off due 
to security concerns. If we disable hyperthreading in the bios and have 
it disabled in the OS then FreeBSD sees one physical and one logical 
processor (from dmesg) and only uses processor 0. If we enable 
hyperthreading in the bios and leave it disabled in the OS it will show 
4 CPUs but only use 0 and 2. Top will show that there is 50% idle CPU 
despite the fact that the box is 100% loaded, CPU 1 and 3 are idle. We 
would expect that FreeBSD would not see logical processors when 
hyperthreading was disabled in either the BIOS or the OS. This may just 
be a communication problem between the BIOS and FreeBSD, but we don't 
see this behavior on other supermicro servers with hyperthreading.

VMSTAT, NETSTAT, NFSSTAT and FSTAT show similar numbers between both 
servers, certainly nothing that would explain why a single httpd process 
requires 20% of a CPU on the 6.2 box and only 5-7% on the 4.11, but we 
could easily be missing something.  We suspected NFS or disk 
bottlenecks, but ran IOZONE tests and found that the 6.2 box is actually 
having better performance on nfs and disk access. We are running a 
slightly customized SMP kernel with device polling enabled. The only 
bottleneck apears to be CPU usage, which works fine on 4.11.

 From what we've read we should not be seeing these performance problems 
with 6.2. So what are we missing? We assume its something stupid that 
will fix this problem quickly and easily, but so far, despite all the 
resources, we have been unable to find a problem with enough in common 
with our own to suggest possible solutions.

Please Help.

thanks
Steve B

-- 
---
Steven H. Baeighkley - Systems Administrator
Front Range Internet, Inc.
stevenb at frii.com - (970) 212-0756