Advice for finding a leaky Apache (probably PHP) process

[Adam Vande More]

On Sun, Apr 25, 2010 at 2:10 AM, Joe Auty <joe at netmusician.org> wrote:

>  Well, I'm fishing. It is also possible that I'm seeing a denial of service
> attack or something, but the result is my Apache processes ballooning and
> CPU usage for some of my httpd processes going up to around 100%. There are
> several PHP apps running on the server, so it is very hard to pinpoint
> things to one app, which is part of the problem.
>
> I can actually see the memory growth, I can sit and watch top and see my
> memory consumption balloon until the machine swaps and then just grinds to a
> halt. Sometimes it gets so bad that I'm forced to killall -9 httpd just to
> bring the machine back to life.
>
> What are some good techniques for trying to ascertain whether a particular
> web app is being exploited for some sort of attack? Since I had to recompile
> PHP and all of my PHP extensions is there a possibility that a particular
> extension is causing memory consumption to balloon? A long time ago I had an
> attack on a very old version of WordPress. I found this via my Apache
> server-status page, but it was sort of a pure fluke that I did find this.
> Surely there has to be better ways to connect httpd processes to pages that
> are being served?
>
> I wish that the machine was a little more responsive when I get to this
> point so that I can ktrace the processes...
>

well if you're just looking for some general advice, I'll tell you what I do
and you take what you like.

-  I don't use Apache anymore for several reasons.  Speed and configuration
are just two of them.  A couple of the more popular http server alternative
are lighttpd and nginx.  I prefer nginx.  Both very small and very fast
compared to apache.  I didn't believe the difference others claimed until I
ran the benchmarks myself.  Not only where they much faster and lighter than
apache, they were also more reliable especially under load.

-  jails are a lightweight method of isolating insecure apps.  Get to know
them and use them extensively.  PHP apps are well known for this type of
thing, and if you're going to run them it's very wise to make sure you're
safe while doing so.  I like to use sysutils/ezjail to create full jails for
each php app each with it's own install of an http server and php.  The root
host would run a reverse proxy of your choosing and direct http requests to
the appropriate jail.  jails can be assigned cpu sets as well eg if your cpu
has 4 cores, a jail can be bound to 1 or more of them.  jails also have
their own process list so top is not so difficult to decipher(your
unresponsive system could be helped by this and you can also be stricter in
your php.ini mem limits per jail).  This method will use a little more
memory, but easily it's the best time-wise from an administration approach
that I've found if you have a lot of different types of clients.

-  use ports-mgmt/portaudit regularly.

-  for debugging php, if truss and ktrace aren't helping, use valgrind.


-- 
Adam Vande More