Sluggish Apache Server

[Barry Steyn]

Hi guys,

We're having a serious problem here with our live server, it's very 
sluggish all of a sudden. The problem is that Apache is *really* slow 
responding to https requests but still fairly quick on http. We've 
checked and ruled out all of the following:

    * CPU usage is normal and so is memory usage
    * All other system daemons seem to be running just fine
    * smartctl -a on both our disks (gmirror RAID 1) says health test
      PASSED, gmirror status fine, smartd running for a week now with
      nothing in the logs
    * nothing strange in the apache access or error logs
    * restarted apache, stopped jboss, upgraded apache to latest patch
      level, even soft rebooted the box but to no avail
    * nobody has done any upgrades, code changes, physical changes or
      anything else to the box before the problem first manifested itself
    * hosting problems - this problem even occurs when you do a wget on
      the same box with https://localhost/... , in fact then it doesn't
      even get to the SSL handshake as it doesn't get to complain about
      the certificate mismatch

The weird thing is that the first time this happened a week ago, there 
was only jboss/seam (which runs behind apache via mod_proxy_ajp) that 
had an issue with sluggishness, all other https pages worked just fine. 
Our tech time was desperate when nobody senior was available and decided 
to hard reboot (power off and on again) the box after which it acted 
really strangely (with disk errors in the logs, other system daemons 
dying randomly) but eventually came right. A week later, sometime this 
afternoon, the problems reoccurred but this time they are chronic, 
nothing we do seems to help.

So, I keep thinking it must be a hardware problem. Not disk (or maybe it 
is?), then perhaps faulty RAM? I always thought faulty RAM results in 
nasty kernel panics, segfaults and other obvious symptoms but not a 
sluggishness in one particular daemon...

Any ideas?

--
Barry