What is loading my server so much?

Michael Powell nightrecon at hotmail.com
Fri Dec 10 15:22:43 UTC 2010

Matthew Seaman wrote:

> On 09/12/2010 11:31, Laszlo Nagy wrote:
>> Today something happened. Number of http processes went up to 200. As a
>> result, number of connections to database also went up to 200, and the
>> web server is now refusing clients with "Cannot connect to database"
>> messages (coming from PHP).
> This is a classic scenario.  Some burst of traffic causes your apache to
> spawn more child processes than will all fit in RAM at one time.
> Consequently, the system starts to swap.  Swapping kills performance.
> This slows everything down so much that there are always requests
> waiting for apache to process, so apache will never find any idle
> children to kill off.  Result misery.
> The answer is to limit the number of child processes apache will spawn.
> Decide how much of your available RAM you can devote to Apache.  Look at
> top(1) to find the maximum size apache processes grow to.  The ratio of
> those two sizes is the maximum number of apache processes your system
> can support.
> Limiting the total number of apache processes sounds counter-intuitive.
> What happens when you get sufficient traffic that apache maxes out?  Web
> queries will generally be queued up until there's an apache child free
> to handle them.  Generally that will take from a few 10s of milliseconds
> on up -- although if you're regularly getting into a state where your
> webserver takes seconds to answer, then it's time to get more beefy
> hardware.

The quintessential "first-try" for this has historically been to set 
keepalives to "Off". Not a solution, just an interim stop gap. If this makes 
any discernible improvement (however small) it confirms the situation.

What I did was switch to the event mpm and FastCGI. Fewer processes, but 
each process spawns lots of threads . Each of the threads within a process 
can reuse the database connection previously established by a different 
thread. Saves connection create/build-up/tear-down cycles as well as RAM. 

Then throw libmemcached into the mix, if possible. To properly utilize 
memcached your PHP has to have some code inserted so it will talk. This sets 
aside a cached buffer pool of RAM for database connections , similar to the 
connection pool you would find in use by a Java servlet container such as 
Tomcat or Resin. If unable to refactor code, in lieu of this you can enable 
MySQL SQL  data cache. Not as effective, but if you get a lot queries which 
are the same SQL it saves cycles.  The amount of YMMV here would depend upon 
how many queries are repetitive. If each and every query is unique it won't 
do much.

So some fictitious example would be like start with some small number of 
stand-by processess such as 8,16,32, etc. Each process might have a thousand 
threads. When this thousand threads is reached Apache opens another process 
with a thousand more. You would still want to set a max-child/max-thread 
count along the lines you previously described. However, this approach gets 
you more headroom at the PHP data access wall of hanging doom.


More information about the freebsd-questions mailing list