shared mem and panics when out of PV Entries

[Igor Sysoev]

On Tue, 25 Mar 2003, Andrew Kinney wrote:

> I'm going to expose my newbness here with respect to BSD 
> memory management, but could the number of files served and 
> filesystem caching have something to do with the PV Entry usage 
> by Apache?  We've got around 1.2 million files served by this 
> Apache.  Could it be that the extensive PV Entry usage has 
> something to do with that?  Obviously, not all are accessed all the 
> time, but it wouldn't take a very large percentage of them being 
> accessed to cause issues if filesystem caching is in any way 
> related to PV Entry usage by Apache.

I think the most of these PV entries related to forked Apaches.
As far as I know PV entries used for files only if these files are mmap()ed.

> I remember reading somewhere (sorry, didn't keep track of the link) 
> that someone running a heavily used Squid proxy had a very 
> similar issue with running out of PV Entries as they got more and 
> more files in the cache.  Squid is basically a modified Apache with 
> proxy caching turned on.

No, Squid is not modified Apache. It's completely different animal.
Apache uses preforked model while Squid uses select()/poll() in the single
process.

> > I think you should try to decrease memory that shared between Apache
> > processes. If you can not change scripts then the single method is to
> > decrease number of Apache processes while keeping to handle the
> > current workload: 1) disable keepalive if it enabled; 2) set Apache
> > behind reverse-proxy server that frees Apache processes
> >    as soon as proxy get the whole responses.
> 
> We had keepalive set to the default of "on" (at least default for this 
> install) with the default keepalive timeout of 15 seconds.
> 
> Dropping the keepalive timeout down to 3 seconds has 
> dramatically reduced the number of Apache processes required to 
> serve the load.  With the new settings, we're averaging 30 to 80 
> Apache processes, which is much more manageable in terms of 
> memory usage, though we weren't anywhere near running out of 
> physical RAM prior to this.  We're servicing a little over 1000 
> requests per minute, which by some standards isn't a huge amount.

I strongly recommend to disable keepalive at all.  Keepalive mostly
make a sense only to download HTML with the inline images.  With heavy
mod_perl server it's simply waste of the resources - megabytes of
a physical memory (usually) or PV entries (in your rare case).

Furthermore heavy Apache processes should not handle images or another
static files.  Heavy Apache should generate dynamic responses only
and should be set behind reverse-proxy (though not mod_proxy).

Let's see a modem user that gets 50K response.  FreeBSD 4.4+ default
in-kernel TCP send buffer is 32K and Apache would quickly fill it
and need to wait for the freeing space for the rest 18K.  A modem user
gets 18K for 6 seconds at 3K/s.  So while mod_perl can generate
answer in 0.1 seconds it needs to wait 6 seconds for transfer this answer.
All your megabytes and PV entries are bound to this transer.
But this long transfer can be made by reverse-proxy that gets response
from heavy server in 0.1 seconds or so.

Furthermore if responses is small enough to fill completely in in-kernel
TCP buffer there is Apache lingering close 2-second timeout.
Before closing the connection Apache calls shutdown(SHUT_WR) and
then calls select() with 2 seconds timeout.  Slow client would not
close the connection on its side because it's still getting response
so Apache waits these 2 seconds and then close() the connection.

So you spend at least 2 seconds for a slow client even your server
can generate responses in a fraction of a second.

> We're still seeing quite heavy PV Entry usage, though.  The 
> reduced number of Apache processes (by more than half) doesn't 
> seem to have appreciably reduced PV Entry usage versus the 
> previous settings, so I suspect I may have been wrong about 
> memory sharing as the culprit for the PV Entry usage.  This 
> observation may just be coincidence, but the average PV Entry 
> usage seems to have gone up by a couple million entries since the 
> changes to the Apache config.

If you have 200M shared memory it takes about 50,000 PV entries per process.
20 processes takes 1 million PV entries.

> Time will tell if the PV Entries are still getting hit hard enough to 
> cause panics due to running out of them.  They're supposed to get 
> forcibly recycled at 90% utilization from what I see in the kernel 
> code, so if we never get above 90% utilization I guess I could 
> consider the issue resolved.
>
> What other things in Apache (besides memory sharing via PHP 
> and/or mod_perl) could generate PV Entry usage on a massive 
> scale?

The file mmap()s. But they should not descreased because they alwayes
shared as compared with copy-on-write fork()ed pages.


Igor Sysoev
http://sysoev.ru/en/