serious performance problems with 6.2 Release

[Lowell Gilbert]

"Steven H. Baeighkley" <stevenb at frii.com> writes:

> Greetings,
>
> We are having some bizarre performance problems on a freshly installed
> 6.2 Release server. This is a supermicro superserver 6022c dual 2.0
> Xeon with 2GB RAM. These CPUs do support hyperthreading. We have done
> significant testing with both hyperthreading turned on and off in the
> bios and in the OS, to no avail.
>
> The server is configured as a web server with apache 2.2.4 php 5.2.0
> and ZendOptimizer. We are running proftpd 1.3.1rc1 and perl 5.8.8. We
> have another server running 4.11 with the same exact hardware and
> software versions. We have updated to the newest bios that Supermicro
> provides.
>
> The trouble is that the 6.2 box performs significantly worse than the
> 4.11 server. The load on the 6.2 server is regularly between 2.0 and
> 6.0. The load on the 4.11 server is between .57 and 1 despite often
> servicing more connections.
>
> We began this process to upgrade into the 6 tree because 4 is EOL. We
> kept the old 4.11 drive from this machine and when replacing it into
> the box performance is excellent just like our other 4.11 box. We have
> tired multiple tuning variables as recommended by both FreeBSD and
> apache and tried the recommendations in the 6.2 errata as well. The
> 6.2 errata states that kern.ipc.nmbclusters="0" will help the kernel
> memory allocator properly deal with high network traffic. We tried
> this and initially thought that the box was showing wonderful
> performance, but then we realized that the box was not allowing much
> network access at all. A single ssh and proftpd connection were all it
> would accept. Apache wouldn't even start giving a MaxClients
> error. Removing this option returned it to functional though poor
> performance mode. Are we missing something with how to use this
> variable? IS this expected behavior?
>
> This particular hardware does display some oddities on both machines,
> running either 6.2 or 4.11. We know that FreeBSD has hyperthreading
> turned off by default. We have done some additional testing with
> hyperthreading turned on in the OS, but we wish for it to remain off
> due to security concerns. If we disable hyperthreading in the bios and
> have it disabled in the OS then FreeBSD sees one physical and one
> logical processor (from dmesg) and only uses processor 0. If we enable
> hyperthreading in the bios and leave it disabled in the OS it will
> show 4 CPUs but only use 0 and 2. Top will show that there is 50% idle
> CPU despite the fact that the box is 100% loaded, CPU 1 and 3 are
> idle. We would expect that FreeBSD would not see logical processors
> when hyperthreading was disabled in either the BIOS or the OS. This
> may just be a communication problem between the BIOS and FreeBSD, but
> we don't see this behavior on other supermicro servers with
> hyperthreading.
>
> VMSTAT, NETSTAT, NFSSTAT and FSTAT show similar numbers between both
> servers, certainly nothing that would explain why a single httpd
> process requires 20% of a CPU on the 6.2 box and only 5-7% on the
> 4.11, but we could easily be missing something.  We suspected NFS or
> disk bottlenecks, but ran IOZONE tests and found that the 6.2 box is
> actually having better performance on nfs and disk access. We are
> running a slightly customized SMP kernel with device polling
> enabled. The only bottleneck apears to be CPU usage, which works fine
> on 4.11.
>
> From what we've read we should not be seeing these performance
> problems with 6.2. So what are we missing? We assume its something
> stupid that will fix this problem quickly and easily, but so far,
> despite all the resources, we have been unable to find a problem with
> enough in common with our own to suggest possible solutions.

Wow.  Let me step back for a moment to appreciate how good this post
is; wonderful stuff to work with in trying to help...

The first thing I would check would be whether the httpd software on
both installations is the same.  I know that I have had trouble
remembering to migrate port configurations on system upgrades, so
maybe other people have the problem too.

Have you checked system processes?  The '-S' option is the way to get
top(1) to show them to you.  That often gives a hint when resources
are vanishing.

What happens if you disable polling?  And what kind of network
interface are you using on the problematic machine?  One thing that
occurs to me is that a lack of DMA on the HTTP packets (to and/or from
the NIC) could produce symptoms like this.

I hope that something here is helpful for you.

Good luck.