collecting pv entries -- suggest increasing PMAP_SHPGPERPROC

[Andrew Kinney]

On 6 May 2003, at 9:55, Bogdan TARU wrote: 

> /kernel: pmap_collect: collecting pv entries -- suggest increasing
> PMAP_SHPGPERPROC
> 
>  The machine being more than 200km away, I cannot compile a new kernel
> right away. So I was wondering if there are any tweaks that I can do
> 'remotely', to solve the problem at least temporary.

We ran into this problem on a similar system to your setup.  Here 
is what we learned: 

PV Entries are directly related the number of simultaneous children 
processes Apache is allowed to spawn. 

To figure out how many PV entries your system is using per 
Apache process, run this command: 

ipcs -a ; sysctl vm.zone | grep PV 

Take a look at the "shared" memory section and find the NATTCH 
value for Apache.  That number is the total number of Apache 
processes that have attached to that shared memory segment. 

The last line of output is your PV entry stats.  The third number in 
from the left is the number of PV entries currently in use. 

Since there are other things using PV entries on the system 
(usually very few - Apache is the big consumer on a web server), 
this isn't an exact science, but take the number of PV entries in 
use and divide it by the number of Apache processes.  That will 
give you a rough estimate of how many PV entries are in use per 
Apache process. 

I don't fully understand the code (pageout daemon) that calls the 
code that generates that warning, but in practical use, the warning  
usually means that you don't have enough pages on the PV entry 
"free" list and it has to compete with other page consumers to get 
more pages from the system.  

Generally, it isn't a serious problem unless you see 5 of those 
messages since the last boot or you're seeing panics from running 
out of PV entries.  Five is the maximum warnings you'll see of that 
type.  Any activity after that that would have generated the warning 
has the warning suppressed, so if you hit five of those messages 
from boot to present time, you have a problem that needs to be 
addressed. 

To address the problem, you'll need to tweak your Apache config 
and restart Apache.  Decrease KeepAliveTimeOut or turn off 
KeepAlive.  Set MaxClients to a number that will prevent your 
system from being crashed by having too many PV entries in use 
and yet still handle the load after reducing KeepAliveTimeout. 

To determine MaxClients, take your maximum PV entries and 
divide it by the number of PV entries per Apache process that you 
calculated above.  Then, subtract a few clients from the result (10 
or 20) and that should be a good number for MaxClients. 

The Apache status page will be your friend when tweaking your 
Apache config so that it efficiently handles the load without hitting 
MaxClients.  It's off by default, so you'll have to turn it on in your 
Apache config if it isn't already on. 

To view your current maximum for PV entries, look at 'sysctl 
vm.zone' and look at the "limit" column for PV ENTRY. 

Determining the maximum PV entries your system will support 
after tweaking PMAP_SHPGPERPROC isn't the easiest of tasks, 
but here's the basic idea: 

pv_entry_max = (shpgperproc * maxproc) + (((((availmem - 1) * 2) + 
1) / pagesize) - firstpage) 

Availmem can be found by doing a 'less /var/run/dmesg.boot' and 
looking at the available memory figure in bytes (not Kbytes).  
Pagesize is typically 4096 bytes on FreeBSD 4.7 i386.  I'm not 
certain on this, but I believe firstpage is the same as pagesize.  
Maxproc can be found with a 'sysctl kern.maxproc'.  Shpgperproc 
defaults to 200 on FreeBSD 4.7, but may be higher depending on 
your kernel config. 

Just remember that you can't set PMAP_SHPGPERPROC 
arbitrarily high or you could overflow your KVA without ever hitting 
the limit for PV ENTRY.  It's best to nudge it upwards gradually 
until the "collecting PV entries" goes away, provided you've 
tweaked Apache as suggested above. 

I know it's kludgy and annoying, but it's just what us heavy Apache 
users have to deal with to avoid crashes due to the state of the 
FreeBSD VM system.  The general idea is that FreeBSD is 
designed for systems that don't do a lot of memory sharing 
between overgrown parent and children processes. 

Here's an article I found awhile back by Matthew Dillon (FreeBSD 
VM guru) that outlines the issue with FreeBSD and memory 
sharing between large numbers of large processes: 

http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/x106.html

Most of these types of problems will go away when Apache 2.0 
becomes useable by most people (missing many vital modules at 
the present time) due to it's threading model (vs. spawning 
children).  For now, we've just got to deal with it, unless someone 
with the right expertise is willing to create a kernel config option 
that changes how PV Entries are allocated for systems that have 
many big fat memory sharing processes. 

Sincerely,
Andrew Kinney
President and
Chief Technology Officer
Advantagecom Networks, Inc.
http://www.advantagecom.net