serious performance problems with 6.2 Release
Steven H. Baeighkley
stevenb at frii.com
Thu Feb 15 00:32:30 UTC 2007
We are having some bizarre performance problems on a freshly installed
6.2 Release server. This is a supermicro superserver 6022c dual 2.0 Xeon
with 2GB RAM. These CPUs do support hyperthreading. We have done
significant testing with both hyperthreading turned on and off in the
bios and in the OS, to no avail.
The server is configured as a web server with apache 2.2.4 php 5.2.0 and
ZendOptimizer. We are running proftpd 1.3.1rc1 and perl 5.8.8. We have
another server running 4.11 with the same exact hardware and software
versions. We have updated to the newest bios that Supermicro provides.
The trouble is that the 6.2 box performs significantly worse than the
4.11 server. The load on the 6.2 server is regularly between 2.0 and
6.0. The load on the 4.11 server is between .57 and 1 despite often
servicing more connections.
We began this process to upgrade into the 6 tree because 4 is EOL. We
kept the old 4.11 drive from this machine and when replacing it into the
box performance is excellent just like our other 4.11 box. We have tired
multiple tuning variables as recommended by both FreeBSD and apache and
tried the recommendations in the 6.2 errata as well. The 6.2 errata
states that kern.ipc.nmbclusters="0" will help the kernel memory
allocator properly deal with high network traffic. We tried this and
initially thought that the box was showing wonderful performance, but
then we realized that the box was not allowing much network access at
all. A single ssh and proftpd connection were all it would accept.
Apache wouldn't even start giving a MaxClients error. Removing this
option returned it to functional though poor performance mode. Are we
missing something with how to use this variable? IS this expected behavior?
This particular hardware does display some oddities on both machines,
running either 6.2 or 4.11. We know that FreeBSD has hyperthreading
turned off by default. We have done some additional testing with
hyperthreading turned on in the OS, but we wish for it to remain off due
to security concerns. If we disable hyperthreading in the bios and have
it disabled in the OS then FreeBSD sees one physical and one logical
processor (from dmesg) and only uses processor 0. If we enable
hyperthreading in the bios and leave it disabled in the OS it will show
4 CPUs but only use 0 and 2. Top will show that there is 50% idle CPU
despite the fact that the box is 100% loaded, CPU 1 and 3 are idle. We
would expect that FreeBSD would not see logical processors when
hyperthreading was disabled in either the BIOS or the OS. This may just
be a communication problem between the BIOS and FreeBSD, but we don't
see this behavior on other supermicro servers with hyperthreading.
VMSTAT, NETSTAT, NFSSTAT and FSTAT show similar numbers between both
servers, certainly nothing that would explain why a single httpd process
requires 20% of a CPU on the 6.2 box and only 5-7% on the 4.11, but we
could easily be missing something. We suspected NFS or disk
bottlenecks, but ran IOZONE tests and found that the 6.2 box is actually
having better performance on nfs and disk access. We are running a
slightly customized SMP kernel with device polling enabled. The only
bottleneck apears to be CPU usage, which works fine on 4.11.
From what we've read we should not be seeing these performance problems
with 6.2. So what are we missing? We assume its something stupid that
will fix this problem quickly and easily, but so far, despite all the
resources, we have been unable to find a problem with enough in common
with our own to suggest possible solutions.
Steven H. Baeighkley - Systems Administrator
Front Range Internet, Inc.
stevenb at frii.com - (970) 212-0756
More information about the freebsd-questions